API (Application Program Interface) automatic feedback system, method and equipment based on large language model

Abstract

The invention relates to an API (Application Program Interface) automatic feedback system, method and equipment based on a large language model. The system comprises a static scanning assembly and a dynamic analysis assembly. And the static scanning component is used for responding to an error existing before the API request is sent to the server, detecting a preset error type to generate static feedback information of the API request according to the received API generated by the large language model API generation unit, and returning the static feedback information to the large language model API generation unit. And the dynamic analysis component is used for responding to an error after the API request is sent to the server, performing real-time information reasoning by utilizing an API document according to the received API generated by the large language model API generation unit and an execution result returned by the server, generating dynamic feedback information for executing the API request, and returning the dynamic feedback information to the large language model API generation unit. By adopting the method, the API request generation precision is remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to an API automatic feedback system, method and device based on a large language model. Background Technology

[0002] Large Language Models (LLMs) are becoming increasingly intelligent and autonomous, evolving towards solving practical real-world tasks and demonstrating impressive intent understanding capabilities. In practice, LLMs are often combined with external tools, such as Application Programming Interfaces (APIs) and code interpreters, further expanding their application scenarios.

[0003] Despite substantial progress in research support and software development, tool-augmented large language models (LLMs) have made significant strides in integrating with external tools. The inherent potential of LLMs with external tools remains largely untapped. Even one of the most state-of-the-art models (GPT-4) struggles to use tools to solve real-world problems. One major issue is the tendency of LLMs to generate illusions—seemingly plausible but unfounded information. This presents a significant challenge when using external tools, which typically require standardized input formats and accurate parameter values. Fine-tuning on specific tool datasets has proven effective in significantly reducing the illusion of using external tools in large models. Existing research often enhances LLMs by providing cue words or external information. Cueing mechanisms such as task demonstrations, self-thoughts, and self-examination are designed to stimulate the planning and reasoning abilities of LLMs. Methods like ReAct and RAG interact with the external environment, providing authentic feedback to guide LLMs in completing tasks.

[0004] Fine-tuning methods on datasets are only suitable for specific tools found in those datasets and are difficult to generalize to a wide range of real-world tools. Furthermore, constructing such datasets incurs high manual costs, hindering widespread adoption. Methods that stimulate LLM capabilities through cue mechanisms can be considered static feedback (not directly interacting with the external environment). They rely on the LLM's own intelligent output rather than providing factual information such as API call format errors or runtime execution errors, which may exacerbate the illusion of power in LLMs. Conversely, methods that interact with the external environment can be considered dynamic feedback methods, but their feedback often lacks sufficient information to accurately guide the LLM in completing tasks. Summary of the Invention

[0005] Therefore, it is necessary to provide an API automatic feedback system, method, and device based on a large language model that can improve the accuracy of API request generation, addressing the aforementioned technical problems.

[0006] An API automatic feedback system based on a large language model, the system comprising: a server, a large language model API generation unit, a static scanning component, and a dynamic analysis component.

[0007] The server acts as a proxy for the real servers corresponding to the collected APIs. It maintains a routing table for an API and its corresponding real server, receives API requests and forwards them to the addresses of the real servers, and integrates the execution results returned by the real servers into API responses.

[0008] The large language model API generation unit is used to parse instruction information and API documentation, output the API request corresponding to the instruction information, and calibrate the information content of the API request based on the received static and dynamic feedback information.

[0009] The static scanning component is used to detect preset error types and generate static feedback information for the API request when there are errors before the API request is sent to the server for execution. This static feedback information is then sent back to the large language model API generation unit.

[0010] The dynamic analysis component is used to respond to API requests sent to the server for execution and when errors occur, it uses the API generated by the large language model API generation unit and the execution result returned by the server to perform real-time information reasoning using the API documentation, generate dynamic feedback information for executing the API request, and send the dynamic feedback information back to the large language model API generation unit.

[0011] An automatic API feedback method based on a large language model, the method comprising: API requests are generated based on a large language model.

[0012] The API request undergoes primary error detection in the local error type document using a static scanning component. The first feedback message is output. If the first feedback message indicates an error, the API request, instruction data, and API document are fed back to the large language model for a calibration loop. Based on preset hyperparameters, the true API request is obtained. Conversely, the server executes the API request, instruction data, and API document, generates a response message, and inputs this response message into a dynamic analysis component for secondary error detection. The second feedback message is output. If the second feedback message indicates an error, dynamic feedback message is generated based on the response message and API document. This dynamic feedback message is fed back to the large language model for a secondary calibration loop. Based on preset hyperparameters, the true API request is obtained. Conversely, the true API request is output based on the response message.

[0013] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps: API requests are generated based on a large language model.

[0014] The API request undergoes primary error detection in the local error type document using a static scanning component. The first feedback message is output. If the first feedback message indicates an error, the API request, instruction data, and API document are fed back to the large language model for a calibration loop. Based on preset hyperparameters, the true API request is obtained. Conversely, the server executes the API request, instruction data, and API document, generates a response message, and inputs this response message into a dynamic analysis component for secondary error detection. The second feedback message is output. If the second feedback message indicates an error, dynamic feedback message is generated based on the response message and API document. This dynamic feedback message is fed back to the large language model for a secondary calibration loop. Based on preset hyperparameters, the true API request is obtained. Conversely, the true API request is output based on the response message.

[0015] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor: API requests are generated based on a large language model.

[0016] The API request undergoes primary error detection in the local error type document using a static scanning component. The first feedback message is output. If the first feedback message indicates an error, the API request, instruction data, and API document are fed back to the large language model for a calibration loop. Based on preset hyperparameters, the true API request is obtained. Conversely, the server executes the API request, instruction data, and API document, generates a response message, and inputs this response message into a dynamic analysis component for secondary error detection. The second feedback message is output. If the second feedback message indicates an error, dynamic feedback message is generated based on the response message and API document. This dynamic feedback message is fed back to the large language model for a secondary calibration loop. Based on preset hyperparameters, the true API request is obtained. Conversely, the true API request is output based on the response message.

[0017] The aforementioned API automatic feedback system, method, and device based on a large language model mainly consist of a server, a large language model-generated API request unit, a static scanning component (SSC), and a dynamic analysis component (DAC). The SSC and DAC respectively implement static and dynamic feedback loops for erroneous API requests generated by the LLM. First, the problems of existing technologies are summarized, identifying the main error types in LLM-generated API requests and incorporating them as factual information into the static feedback, effectively alleviating the problem of illusions caused by the lack of factual information in traditional static feedback methods. The fine-grained static feedback in this system narrows down the sources of errors, facilitating accurate correction of erroneous API requests by the LLM. Dynamic feedback automates the API documentation retrieval process, using error information in the API response as an index to query related error causes in the API documentation, enhancing the information content of dynamic feedback. This effectively alleviates the problem of traditional dynamic feedback methods failing to stimulate the LLM's ability to correct API request errors due to insufficient information. Simultaneously, the combined static and dynamic feedback loops can promote the generation of correct API requests, achieving performance far exceeding that of single feedback and significantly reducing the overhead of interacting with the LLM to complete tasks. Finally, for issues that cannot be resolved through automatic feedback, the AutoFeedback framework provides a human-computer assistance interface. Users can refer to the logs generated during the feedback process and manually provide more specific feedback or other methods. Attached Figure Description

[0018] Figure 1 This is a block diagram of an API automatic feedback system based on a large language model in one embodiment; Figure 2 This is a flowchart illustrating an API automatic feedback method based on a large language model in one embodiment. Figure 3 This is a schematic diagram of a static feedback template for automatic API feedback based on a large language model in one embodiment. Figure 4 This is a schematic diagram of the API automatic feedback and dynamic feedback process based on a large language model in one embodiment; Figure 5 This is a schematic diagram of the design flow of AutoFeedback, an LLM-based API request generation framework, in one embodiment. Figure 6 This is a schematic diagram of a feedback-aware adaptive large language model API execution request framework in one embodiment. Figure 7 This is a schematic diagram of the operation code for re-planning prompt words for new candidate APIs in one embodiment. Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0020] In one embodiment, such as Figure 1 As shown, an API automatic feedback system based on a large language model is provided, including: a server 102, a large language model API generation unit 104, a static scanning component 106, and a dynamic analysis component 108, wherein: Server 102 acts as a proxy for the real servers corresponding to the collected APIs. It maintains a routing table for an API and its corresponding real server, receives API requests and forwards them to the addresses of the real servers, and integrates the execution results returned by the real servers into API responses.

[0021] The large language model API generation unit 104 is used to parse instruction information and API documentation, output the API request corresponding to the instruction information, and calibrate the information content of the API request based on the received static feedback information and dynamic feedback information.

[0022] The static scanning component 106 is used to detect preset error types and generate static feedback information for the API request based on the API generated by the large language model API generation unit when there is an error before the API request is sent to the server for execution, and then send the static feedback information back to the large language model API generation unit.

[0023] It is worth noting that the static scanning component performs a local scan before the API request generated by the large language model is sent to the server for execution. It matches preset error types and generates static feedback, which is then fed back to the large model.

[0024] The dynamic analysis component 108 is used to respond to API requests sent to the server for execution and when errors occur, it uses the API generated by the large language model API generation unit and the execution result returned by the server to perform real-time information reasoning using the API documentation, generate dynamic feedback information for executing the API request, and send the dynamic feedback information back to the large language model API generation unit.

[0025] It is worth noting that the dynamic analysis component obtains the execution result after the API request generated by the large language model is sent to the server for execution, and then retrieves the detailed error content of the execution result by searching the API documentation, forming dynamic feedback, which is then fed back to the large model.

[0026] In one embodiment, the preset error types include: E1 sub-errors, E2 sub-errors, E3 sub-errors, and E4 sub-errors. E1 sub-errors include: the content generated by the large language model API generation unit cannot be parsed as the API. E2 sub-errors include: the large language model API generation unit calls an incorrect API name. E3 sub-errors include: the large language model API generation unit calls an incorrect parameter name. E4 sub-errors include: the parameter type called by the large language model API generation unit is inconsistent with the API documentation.

[0027] It's worth noting that if the LLM (Large Language Model) cannot generate the correct API format, it is categorized as E1. E2 API name errors primarily originate from three dimensions: E2. Selection error: calling other candidate API names; E2.2 Character error: incorrect capitalization in the API name, or different naming conventions (e.g., camelCase, underscore); E2.3 Semantic error: semantically similar to the correct API name. Similar to E2, E3 parameter errors also originate from these three dimensions: E3.1 Selection error: calling parameter names in other candidate APIs; E3.2 Character error: incorrect capitalization in the parameter name, or different naming conventions; E3.3 Semantic error: semantically similar to the correct parameter name. Rule-based methods struggle to pinpoint the error type of parameter values, leading to the introduction of E4.1 Type error: parameter type inconsistent with the API documentation. These predefined error types will be used as pseudo-facts in the feedback, significantly reducing the illusions the LLM experiences when generating API requests.

[0028] In one embodiment, the static scanning component 106 includes an API retriever and an automatic error detection unit. The static scanning component is further configured to receive instruction information and API documentation; the API retriever recommends API names matching the instruction information; and the automatic error detection unit locates the E2 sub-errors corresponding to the API request to generate static feedback information.

[0029] It's worth noting that the API retrieval tool recommends API names from the API documentation that best match the instruction information, forming a recommended API subset. It only detects E2-class errors; other error classes are detected by parsing the API request and comparing it with the API documentation. The component inputs are as follows: the Static Scanning Component (SSC) input includes user instructions, API documentation (including API name and description, required parameter types and descriptions, exception descriptions, etc.), and API requests in the format of APINAME (key1=value1, key2=value2, ...) generated by the LLM. Furthermore, the Dynamic Analysis Component (DAC) input also includes API responses from the API server.

[0030] In one embodiment, the dynamic analysis component 108 includes a document retrieval unit and a dynamic feedback unit. The dynamic analysis component is further configured to receive feedback information after an API request is executed on the server. The document retrieval unit uses a Retrieval Augmented Generation (RAG) algorithm to extract error information from the feedback information, vectorizes the extracted error information using an embedding model, and retrieves a dynamic error text database built based on the API document using the output vector as an index. The component also uses the ReAct algorithm to infer factual information about the API request response from the large language model API generation unit based on the feedback information, matches the factual information with the dynamic error text database, and generates dynamic feedback information.

[0031] It's worth noting that the dynamic error text database is built according to the API documentation and API hierarchy. During runtime, the input API response is fed into the embedding model to form an index vector. This vector is then used to retrieve the dynamic error text database, finding the text most relevant to the API response. This expands the original API response, resulting in a detailed API error description and dynamic feedback. DAC (Data Access Detection) focuses on situations where many APIs return insufficient information (such as error codes) after execution on the server, rather than detailed error messages. In such cases, users must continue searching the API documentation for detailed explanations of the corresponding error codes. This process often consumes a significant amount of user time. DAC automates this retrieval, improves the detail of dynamic feedback, and helps correct errors in API requests.

[0032] Specific limitations regarding the API autofeedback device based on large language models can be found in the limitations of the API autofeedback method based on large language models mentioned above, and will not be repeated here. Each module in the aforementioned API autofeedback device based on large language models can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0033] In one embodiment, such as Figure 2 As shown, an automatic API feedback method based on a large language model is provided, which is then applied to... Figure 1 Taking the feedback system in the example, the following steps are included: Step 202: Generate API requests based on the large language model.

[0034] Specifically, LLM takes user instructions as input and generates API requests based on the API documentation.

[0035] Step 204: The API request undergoes first-level error detection in the local error type document using the static scanning component, outputting first feedback information. If the first feedback information is incorrect, the API request, instruction data, and API document are fed back to the large language model for API request calibration loop. Based on preset hyperparameters, the true API request is obtained. Conversely, the server executes the API request, instruction data, and API document, generates response information, and inputs the response information into the dynamic analysis component for second-level error detection, outputting second feedback information. If the second feedback information is incorrect, dynamic feedback information is generated based on the response information and API document, and this dynamic feedback information is fed back to the large language model for API request second-level calibration loop. Based on preset hyperparameters, the true API request is obtained. Conversely, the true API request is output based on the response information.

[0036] There are common problems with LLM-based API request generation, and four main types of errors are summarized: E1: NO_API: No API request was generated, or the API request format was incorrect, causing parsing failure; E2: API_NAME_MISMATCH: The generated API name does not match the format of the API documentation or the intent of the user instruction; E3: PARAMETER_INVALID: The generated output contains invalid parameter names that do not appear in the API documentation; E4: INPUT_MISMATCH: The type of the parameter value does not match the API documentation or the user instruction.

[0037] Specifically, a deep learning model is used as a sub-component of SSC—the API retrieval unit M—which can calculate the semantic similarity between two texts:

[0038] Model M encodes user instructions and API descriptions in a document into two embeddings and calculates their relevance through embedding similarity. Then, APIs in the document are ranked according to the similarity score between the description and the instruction. SSC lists the k APIs with the highest scores (usually k is set to 1) as the relevant API set R, where R is a subset of API documents where API names are considered to meet user needs and will be provided to the subsequent error detection subcomponent.

[0039] Furthermore, the LLM output may not contain API requests or may be unparseable, which can be summarized as E1. For other generated API requests, SSC performs the following automatic error detection: API name (E2.1, E2.2, E2.3). First, SSC determines the API request generated by LLM. Is it in If not ( If the API name appears in the API documentation, the sub-error type is E2.1. This is considered a misunderstanding of the user's intent by the LLM, incorrectly selecting a different API. Due to different coding styles, the LLM may have incorrectly generated an underscore name (e.g., user_login), whose true value should be a camelCase method name (userLogin). SSC changes all API names to lowercase and removes characters other than alphanumeric characters using a regular expression (r'[a-zA-Z]'). After regularization, if the API name appears in the API documentation, the sub-error type is E2.1. and If the names are the same, the sub-error type is E2.2. Detection of E2.3 primarily targets LLMs that fabricate non-existent or forged APIs in their illusions. For example, a user instruction might be "I'm trying to find out how much aspirin is left.", generating a fake API as "find_aspirin_number()", while the real API is "list_medicines(name='aspirin')". Also using... To test The semantic similarity score between the name of the API and the names of candidate APIs in the API documentation. If there is an API If the score exceeds the threshold (usually set to 0.5), the sub-error type is E2.3. Other cases are "other" errors in E2.

[0040] API parameter names (E3.1, E3.2, E3.3). When At this point, the next step for SSC is to verify the correctness of the API parameter names. Unlike the API name, the correct parameter names are already provided in the API documentation. If If the parameter name in the API documentation does not exist in the corresponding description, it is considered an E3 error. Therefore, this incorrect API parameter is considered... If other API parameter lists appear in the documentation... If the name is specified, the sub-error type is E3.1. SSC's operation on API parameter names is the same as in E2.2 and E2.3. (Regularization modification follows.) Parameters in other APIs When matching, the sub-error type is E3.2. If the parameters in the API documentation... and If the name similarity score is greater than the threshold, it is considered an E3.3 error. Other cases are "other" errors in E3.

[0041] API parameter values ​​(E4.1). Due to the high flexibility of parameter values, a preliminary judgment can only be made based on the parameter type. SSC parsing. All parameter values ​​are processed and compared with the data types specified in the API documentation. When parameter values... Matching error, corresponding parameter is In such cases, the sub-error type is E4.1. In some situations, no errors are found after static scanning, but they exist during the call process, or the call result still does not meet the user's requirements; in these cases, it is an "other" error under E4.

[0042] Return values. SSC detects errors in ascending numerical order (e.g., E1→E2, E2.2→E2.3). Once an error is detected, it returns the corresponding value to proceed to the next step of static feedback. Besides returning the error type, other return values ​​are shown in Table 1.

[0043] Table 1 Return values ​​during the error detection phase

[0044] 3) Static Feedback: The SSC receives the generated API request as input, performs error detection with the assistance of the API retrieval tool, and generates static errors. Then, based on the error type, the SSC selects the corresponding prompt template and fills it in according to the given format based on other return values ​​to generate static feedback. For example... Figure 3 As shown, the template is roughly divided into five parts, with the feedback granularity gradually increasing. First, it declares an error in the API request; second, it indicates the location and content of the error; third, it eliminates the source of the error that has already passed detection (for E2.3, this excludes E2.1 and E2.2); fourth, it identifies the source of the error and provides corrective suggestions; fifth, it requests the LLM to regenerate the API request. Some static feedback for errors may omit certain parts. For example, in E1, because only one detection was performed, the third part is missing from the feedback.

[0045] Furthermore, DAC retrieves detailed information about the corresponding errors from the API documentation. A Retrieval Augmented Generation (RAG) method is used to extract error-related information. First, the API documentation is split to obtain all document content related to the API (including function descriptions, exception explanations, etc.). Then, this content is broken down into sentence-level granularity, and the sentences are vectorized using an embedding model. Next, semantically similar sentences are placed into the same data blocks, achieving semantic segmentation and vectorization. This is equivalent to building a vector-indexed text database, where vectorized sentences serve as the index.

[0046] Furthermore, in Figure 4 In the retrieval process shown, DAC uses API names (such as route_planning) to match corresponding block vectors. Then, the same embedding model is used to vectorize the user query, and the most similar block vector is found as an index (implemented via an approximate nearest neighbor algorithm) to retrieve the corresponding text from the previously built database. Finally, more detailed error information is obtained.

[0047] Furthermore, integrating the ReAct algorithm into the DAC allows the LLM to observe API responses and incorporate factual information into inference. Simultaneously, error messages from the document retrieval subcomponent are combined with the API response to create dynamic feedback, as detailed below: The server executes the sent API request and receives the corresponding API response. The next step is to evaluate whether the user's needs have been met or whether the maximum feedback count has been reached. If the conditions are met, the task ends and exits. Otherwise, it enters a dynamic feedback loop: the DAC first retrieves the corresponding error message from the documentation; the ReAct algorithm treats the API request as an action, combines the API response and error message as an observation, and feeds it back to the LLM along with the historical interaction record; newly generated API requests and thoughts are stored as feedback records along with the action and observation. Finally, the new execution result returned by the server is evaluated.

[0048] In the aforementioned API autofeedback method based on a large language model, the API autofeedback system applied to the large language model implements static and dynamic feedback loops for erroneous API requests generated by the LLM. First, it summarizes the problems of existing technologies, identifies the main error types in LLM-generated API requests, and incorporates these as factual information into the static feedback, effectively alleviating the problem of illusions caused by the lack of factual information in traditional static feedback methods. Furthermore, the fine-grained static feedback in this system narrows down the sources of errors, facilitating the LLM's accurate correction of erroneous API requests. Dynamic feedback automates the API documentation retrieval process, using error information in the API response as an index to query related error causes in the API documentation, enhancing the information content of dynamic feedback. This effectively alleviates the problem of traditional dynamic feedback methods failing to stimulate the LLM's ability to correct API request errors due to insufficient information. Simultaneously, the combined static and dynamic feedback loops promote the generation of correct API requests, achieving performance far exceeding that of single feedback, and significantly reducing the overhead of interacting with the LLM to complete tasks. Finally, for problems that cannot be solved through automatic feedback but are provided by the AutoFeedback framework's human-machine assistance interface, users can refer to the logs generated during the feedback process to manually provide more specific feedback or other methods.

[0049] In one embodiment, the user-input command data is parsed and matched with the local API documentation on the server based on a large language model to generate an API request corresponding to the command data.

[0050] In one embodiment, if the first feedback information is an error, the semantic similarity between the instruction data and the API document is calculated using the deep learning model of the static scanning component, and a subset of APIs to be recommended is obtained based on the semantic similarity. This subset of APIs is then used by the error detection unit of the static scanning component to perform local error type retrieval, outputting the erroneous API requests after the first-level feedback and the corresponding instruction data to the large language model for calibration looping. Based on preset hyperparameters as loop constraints, the true API requests are obtained.

[0051] It's worth noting that the subset of APIs to be recommended by the static scanning component is only used to detect E2 type errors. Other error types are determined by comparing the parsed API with the API documentation. The LLM takes the API documentation and user instructions as input, parses the generated output to obtain the API request. The SSC performs a local static scan to determine if any errors exist in the API request. If none are found, the request continues to be sent; if errors are found, error detection is performed to locate sub-errors, and precise static feedback is provided to the LLM.

[0052] In one embodiment, the response information includes: the API request, the response result of executing the API request, and the instruction data corresponding to the API request. If the second feedback information is an error, the response error information is extracted using the retrieval enhancement generation algorithm employed by the document retrieval unit of the dynamic analysis component. This extracted information is then vectorized using an embedding model. Using the output vector as an index, a dynamic error text database constructed based on the API document is retrieved to obtain detailed API error information. Based on the detailed API error information and the ReAct algorithm's reasoning about the API request's response facts, dynamic feedback information is generated. This dynamic feedback information is fed back to the large language model for a secondary calibration loop of the API request. Using preset hyperparameters as loop constraints, the true API request is obtained.

[0053] It is worth noting that LLMs use their own internal representations to produce outputs that are not based on the external world, and tools that enhance the interaction between LLMs and external tools are crucial, which can mitigate the possibility of factual illusions.

[0054] In one embodiment, such as Figure 5As shown, a design flow for an LLM-based API request generation framework, AutoFeedback, is provided. The AutoFeedback framework contains two loosely coupled complementary components: a Static Scanning Component (SSC) and a Dynamic Analysis Component (DAC). These are combined with the original process of LLM calling the API (gray arrow) to create two feedback loops. To avoid infinite loops, a maximum number of feedback iterations is set and used as a hyperparameter.

[0055] 1) Static Feedback Flow (dashed arrow). The LLM takes the API documentation and user instructions as input, parses the generated output to obtain the API request. The SSC performs a local static scan to determine if there are any errors in the API request. If not, the request continues to be sent; if so, error detection is performed to locate sub-errors and provide precise static feedback to the LLM.

[0056] 2) Dynamic Feedback Process (dotted arrows). After an API request is sent to the server and executed, the user evaluates whether the returned API response meets their needs. For failed API requests, the DAC retrieves detailed error information from the API documentation and combines it with the API response as dynamic feedback.

[0057] It's worth noting that SSC primarily aims to help generate API requests that conform to user instructions and format standards. It's also worth mentioning that SSC cannot provide feedback on the details of parameter values. However, this doesn't mean SSC is without benefit in correcting parameter values; rather, it narrows down the sources of errors, allowing LLMs in both static and dynamic feedback to focus on the parameter values.

[0058] In one embodiment, the dynamic analysis component is further configured to perform a semantic error detection step including semantic verification. Specifically, the steps are as follows: using the large language model API generation unit as an evaluator, evaluating whether the response information conforms to the user instruction based on preset evaluation prompts, and obtaining a discrete score output by the evaluator. If any discrete score output by the evaluator is 0, it is determined that the API request has a semantic error, and the dynamic feedback information generation process is initiated; otherwise, it is determined that the API request has no semantic error, and the true API request is output.

[0059] In one embodiment, the dynamic analysis component, before generating dynamic feedback information, further includes effective parameter value estimation. Specifically, the steps involve: a tool calling a historical database, which consists of several API use cases. Composition, in which, For instructions, For API requests, The execution result is as follows: The API use case includes errors corresponding to preset error types, semantic errors, and the correctly executed API call trajectory. The parameter value of the current API request is compared with the correct API use case in the historical database. If the parameter value does not appear or there is a semantic difference between the parameter value and the API use case, and the semantic difference is greater than a threshold, then the parameter value is determined to be incorrect. If the parameter values ​​of all API requests in the current task exist in the historical database but fail the semantic verification, then the existing API function combination is determined to be unable to meet the user's needs.

[0060] In one embodiment, the dynamic analysis component is further configured to generate dynamic feedback information for executing the API request. Specifically, if the parameter value is determined to be incorrect, a two-stage use case retrieval method is employed. This involves filtering the use cases in the historical database based on the API name, retaining the use cases with the same name as the current API request, and then calculating a similarity score based on the user instruction and execution result.

[0061] in, To score the similarity, For the current trajectory, For API use cases, , For different constants, For the similarity calculation function of the embedded model, a preset number of positive and negative API use cases are retrieved as feedback. For the number of API requests, , For user commands at different times. , This represents the execution results at different times. If the existing API functionality combination cannot meet the requirements, the large language model is driven by preset prompts to re-plan candidate APIs from the API file. Based on the prompts, the large language model analyzes user intent step-by-step, extracts key parameters, compares potential APIs, and demonstrates the rationality of the selection, outputting a structured result containing the reasoning process, the candidate APIs, and the parameter values. The retrieved API use cases or the re-planned candidate APIs are integrated into dynamic feedback information and fed into the large language model for a two-stage calibration loop.

[0062] It is worth noting that, such as Figure 6 As shown, a feedback-aware adaptive large language model API execution request framework is provided, including a compliance error feedback module and a semantic error feedback module. The compliance error feedback module is roughly divided into four parts, with the granularity of the feedback gradually becoming finer: 1. An error was found in the API name / parameters; (Locate the error location) 2. The API name / parameters you incorrectly generated are: <Return values ​​in Table 1>; (Locate the error content) 3. The API name / parameter is incorrect because it does not appear in the API documentation... (Explanation of the error) 4. You should select compliant content from the following APIs / parameters... (provide error correction methods) API requests that pass compliance checks can be executed normally, and the corresponding API server returns the execution result. However, due to semantic errors in parameter values ​​or incomplete functionality of existing API combinations, LLM still cannot fulfill the task requirements. The semantic error feedback module mainly fixes errors that cannot be detected by rule matching, such as 1. inappropriate parameter calls; 2. incoordination between multiple APIs, preventing the completion of task requirements. Details are as follows: 1) Semantic verification After compliance verification, the API request is forwarded to the API server for execution, and the corresponding execution result is subsequently returned. Since compliance verification cannot fully cover "Other" errors in the E4 of API requests, the framework should evaluate semantic errors based on these execution results.

[0063] The module uses evaluation prompts specific to its usage scenario and employs an LLM-as-a-judge approach, using multiple LLMs as evaluators to assess whether the execution result conforms to the user's instructions. Each evaluator LLM is expected to output a discrete score s, where s∈{0,0.5,1} represents whether the user's request was unresolved, uncertain, or resolved, respectively. If any evaluator outputs 0, the API request is considered unsuccessful and categorized as an "other" error in E4. Otherwise, the LLM completes the task successfully without requiring feedback.

[0064] 2) Parameter value effective estimation The "other" error in E4 stems from improper use of APIs by the LLM, which can be categorized into inappropriate parameter calls or incompatibility between multiple APIs, resulting in the inability to fulfill task requirements. To further pinpoint finer-grained errors, the project collected success and failure logs to build an API-level historical interaction log database, including parameter usage experience. The database consists of use cases. Composed of $, where Indicates instructions, Indicates an API request. This indicates the execution result. The basic attributes of the database are shown in Table 2.

[0065] Table 2 Tool Call History Database

[0066] For each generated API request, the framework compares the correct API use cases in the historical database. If a parameter value in the API request has not appeared in the database or has a significantly different semantic meaning, it is considered an incorrect parameter value. If all API requests in the task contain parameter values ​​that have appeared in the database but still fail the semantic check, the existing API request functionality combination is deemed insufficient to meet the user's needs. The framework uses the following operation code: Figure 6 The indicated prompts allow the LLM to re-evaluate new candidate APIs from its existing API database.

[0067] 3) Semantic error feedback construction For incorrect parameter values, the framework implements a two-phase use case retrieval method to obtain similar API examples. Current trajectory By user instructions API requests and their execution results The framework first filters examples based on their API names, retaining only those API names displayed. The example is shown in the image. Next, we perform a search that includes user commands and execution results, with a similarity score of [score missing]. :

[0068] in, To score the similarity, For the current trajectory, For API use cases, , For different constants, For the similarity calculation function of the embedded model, a preset number of positive and negative API use cases are retrieved as feedback. For the number of API requests, , For user commands at different times. , The results are for execution at different times. The feedback constructed from these retrieved API experiences consists of multiple positive and negative API use cases.

[0069] It should be understood that, although Figure 2 , Figure 5 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 2 , Figure 5At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0070] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements an API automatic feedback method based on a large language model. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0071] Those skilled in the art will understand that Figure 1 , Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the computer device to which the present invention is applied. A specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0072] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to perform the following steps: API requests are generated based on a large language model.

[0073] The API request undergoes primary error detection in the local error type document using a static scanning component. The first feedback message is output. If the first feedback message indicates an error, the API request, instruction data, and API document are fed back to the large language model for a calibration loop. Based on preset hyperparameters, the true API request is obtained. Conversely, the server executes the API request, instruction data, and API document, generates a response message, and inputs this response message into a dynamic analysis component for secondary error detection. The second feedback message is output. If the second feedback message indicates an error, dynamic feedback message is generated based on the response message and API document. This dynamic feedback message is fed back to the large language model for a secondary calibration loop. Based on preset hyperparameters, the true API request is obtained. Conversely, the true API request is output based on the response message.

[0074] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: API requests are generated based on a large language model.

[0075] The API request undergoes primary error detection in the local error type document using a static scanning component. The first feedback message is output. If the first feedback message indicates an error, the API request, instruction data, and API document are fed back to the large language model for a calibration loop. Based on preset hyperparameters, the true API request is obtained. Conversely, the server executes the API request, instruction data, and API document, generates a response message, and inputs this response message into a dynamic analysis component for secondary error detection. The second feedback message is output. If the second feedback message indicates an error, dynamic feedback message is generated based on the response message and API document. This dynamic feedback message is fed back to the large language model for a secondary calibration loop. Based on preset hyperparameters, the true API request is obtained. Conversely, the true API request is output based on the response message.

[0076] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0077] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0078] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. An API automatic feedback system based on a large language model, characterized in that, The system includes: a server, a large language model API generation unit, a static scanning component, and a dynamic analysis component; The server acts as a proxy for the real servers corresponding to the collected APIs. It maintains a routing table for the APIs and the real servers corresponding to the APIs, receives API requests and forwards the API requests to the address of the real servers, and integrates the execution results returned by the real servers into API responses. The large language model API generation unit is used to parse instruction information and API documentation, output the API request corresponding to the instruction information, and calibrate the information content of the API request based on the received static feedback information and dynamic feedback information. The static scanning component is used to respond to the API request being sent to the server for execution if there is an error. Based on the API generated by the large language model API generation unit, it detects a preset error type, generates static feedback information for the API request, and sends the static feedback information back to the large language model API generation unit. The dynamic analysis component is used to respond to errors that occur after the API request is sent to the server for execution. Based on the API generated by the large language model API generation unit and the execution result returned by the server, the component performs real-time information reasoning using the API document to generate dynamic feedback information for executing the API request and sends the dynamic feedback information back to the large language model API generation unit.

2. The system according to claim 1, characterized in that, The preset error types include: E1 sub-errors, E2 sub-errors, E3 sub-errors, and E4 sub-errors; The E1 sub-error includes: the content generated by the large language model API generation unit cannot be parsed as the API; The E2 sub-errors include: the name of the API call error in the large language model API generation unit; The E3 sub-errors include: the name of the error parameter in the large language model API generation unit call; The E4 sub-error includes: the parameter type of the large language model API generation unit call is inconsistent with the API documentation.

3. The system according to claim 2, characterized in that, The static scanning component includes: an API retriever and an automatic error detection unit; The static scanning component is also used to receive the instruction information and the API document. The API retrieval tool recommends API names that match the instruction information, so as to locate the E2 sub-error corresponding to the API request through the automatic error detection unit and generate static feedback information.

4. The system according to claim 3, characterized in that, The dynamic analysis component includes: a document retrieval unit and a dynamic feedback unit; The dynamic analysis component is further configured to receive feedback information after the API request is executed on the server. The document retrieval unit uses a retrieval enhancement generation algorithm to extract error information from the feedback information, vectorizes the extracted error information using an embedding model, and retrieves a dynamic error text database built based on the API document using the output vector as an index. It also uses the ReAct algorithm to infer factual information about the large language model API generation unit's response to the API request based on the feedback information, and matches the factual information with the dynamic error text database to generate dynamic feedback information.

5. An API automatic feedback method based on a large language model, characterized in that, The method includes: Generate API requests based on a large language model; The API request undergoes first-level error detection in the local error type document using a static scanning component, outputting first feedback information. If the first feedback information is incorrect, the API request, instruction data, and API document are fed back to the large language model for a calibration loop, and the true API request is obtained based on preset hyperparameters. Conversely, if the first feedback information is incorrect, the server executes the API request, instruction data, and API document, generates response information, and inputs the response information into a dynamic analysis component for second-level error detection, outputting second feedback information. If the second feedback information is incorrect, dynamic feedback information is generated based on the response information and the API document, and the dynamic feedback information is fed back to the large language model for a second-level calibration loop, and the true API request is obtained based on preset hyperparameters. Conversely, if the second feedback information is incorrect, the true API request is output based on the response information.

6. The method according to claim 5, characterized in that, The dynamic analysis component is further used to perform semantic error detection steps, including semantic verification, specifically: The large language model API generation unit is used as the evaluator to evaluate whether the response information conforms to the user instruction according to the preset evaluation prompt words, and obtain the discrete score output by the evaluator. If the discrete score output by any evaluator is 0, it is determined that the API request has a semantic error and the dynamic feedback information generation process is started. Conversely, if no semantic error is found, the API request is determined to have no semantic class error, and the actual API request is output.

7. The method according to claim 6, characterized in that, Before generating dynamic feedback information, the dynamic analysis component also includes effective parameter value estimation, the specific steps of which are as follows: The build tool calls a historical database, which consists of several API use cases. Composition, in which, For instructions, For API requests, The execution result; the API use case includes errors corresponding to preset error types, semantic errors, and the API call trace of correct execution; The parameter value of the current API request is compared with the correct API use case in the historical database. If the parameter value does not appear or there is a semantic difference between the parameter value and the API use case, and the semantic difference is greater than a threshold, then the parameter value is determined to be incorrect. If the parameter values ​​of all API requests in the current task exist in the historical database, but fail the semantic verification, then the existing API function combination is determined to be unable to meet the user's needs.

8. The method according to claim 7, characterized in that, The dynamic analysis component is also used to generate dynamic feedback information for executing the API request, specifically through the following steps: If the parameter value is determined to be incorrect, a two-stage test case retrieval method is used. Test cases in the historical database are filtered based on the API name, and those matching the current API request name are retained. Then, a similarity score is calculated based on the user command and execution result. in, To score the similarity, For the current trajectory, For API use cases, , For different constants, For the similarity calculation function of the embedded model, a preset number of positive and negative API use cases are retrieved as feedback. For the number of API requests, , For user commands at different times. , For the execution results at different times; If the existing API function combination cannot meet the requirements, the large language model is driven by preset prompt words to re-plan candidate APIs from the API file. According to the prompt words, the large language model analyzes the user intent step by step, extracts key parameters, compares potential APIs, and demonstrates the rationality of the selection, and outputs a structured result containing the reasoning process, the candidate APIs, and the parameter values. The retrieved API use cases or re-planned candidate APIs are integrated into dynamic feedback information, which is then fed into the large language model for a two-stage calibration cycle.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 5 to 8.

Images