API (Application Program Interface) automatic feedback system, method and equipment based on large language model

Abstract

The invention relates to an API (Application Program Interface) automatic feedback system, method and equipment based on a large language model. The system comprises a static scanning assembly and a dynamic analysis assembly. And the static scanning component is used for responding to an error existing before the API request is sent to the server, detecting a preset error type to generate static feedback information of the API request according to the received API generated by the large language model API generation unit, and returning the static feedback information to the large language model API generation unit. And the dynamic analysis component is used for responding to an error after the API request is sent to the server, performing real-time information reasoning by utilizing an API document according to the received API generated by the large language model API generation unit and an execution result returned by the server, generating dynamic feedback information for executing the API request, and returning the dynamic feedback information to the large language model API generation unit. By adopting the method, the API request generation precision is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an API automatic feedback system, method and device based on a large language model. Background Art

[0002] Large Language Models (LLMs) are becoming more and more intelligent and autonomous, moving towards solving practical tasks in the real world, and demonstrating impressive intent understanding capabilities. In practical applications, LLMs are often combined with external tools such as application programming interfaces (APIs) and code interpreters, further expanding their application scenarios.

[0003] Although tool-augmented LLMs have made substantial progress in areas such as research assistants and software development, the inherent potential of LLMs to integrate with external tools remains untapped. Even one of the most advanced existing models (GPT-4) is still difficult to use tools to solve real-world problems. One of the main problems is that LLMs are prone to hallucinations - producing seemingly reasonable but unfounded information. This poses a huge challenge to using external tools that usually require standardized format inputs and accurate parameter values. Fine-tuning on tool-specific datasets has been shown to be an effective method to greatly reduce the hallucinations of large models using external tools. Existing studies usually feed back prompt words or external information to LLMs to enhance their capabilities. For example, task demonstrations, self-thoughts, and self-examination design prompt mechanisms to stimulate the planning and reasoning capabilities of LLMs. Methods such as ReAct and RAG interact with the external environment to provide real feedback information to guide LLMs to complete tasks.

[0004] Methods that fine-tune on datasets are only applicable to specific tools that appear in the datasets and are difficult to generalize to a wide range of tools in the real world. In addition, the construction of such datasets has high labor costs and is difficult to promote. Methods that stimulate LLM capabilities by designing prompt mechanisms can be regarded as static feedback methods (no direct interaction with the external environment). They rely on LLM's own intelligent output rather than providing factual information such as API call format errors and runtime execution errors, which may exacerbate LLM's hallucination phenomenon. In contrast, methods that interact with the external environment can be regarded as dynamic feedback methods, whose feedback usually lacks sufficient information and is difficult to accurately guide LLM to complete tasks. Summary of the invention

[0005] Based on this, it is necessary to provide an API automatic feedback system, method and device based on a large language model that can improve the accuracy of API request generation in response to the above technical problems.

[0006] An API automatic feedback system based on a large language model comprises: a server, a large language model API generation unit, a static scanning component and a dynamic analysis component.

[0007] The server acts as a proxy for the real servers corresponding to the collected APIs, and is used to maintain a routing table of APIs and real servers corresponding to the APIs, receive API requests and forward API requests to the address of the real server, and integrate the execution results returned by the real server into API responses.

[0008] The large language model API generation unit is used to parse instruction information and API documents, output API requests corresponding to the instruction information, and calibrate the information content of the API request according to the received static feedback information and dynamic feedback information.

[0009] The static scanning component is used to respond to an error before an API request is sent to the server for execution. According to the API generated by the received large language model API generation unit, it detects the preset error type to generate static feedback information of the API request, and transmits the static feedback information back to the large language model API generation unit.

[0010] The dynamic analysis component is used to respond to an error after an API request is sent to the server for execution. Based on the API generated by the received large language model API generation unit and the execution result returned by the server, the API document is used for real-time information reasoning, and dynamic feedback information of the execution API request is generated, and the dynamic feedback information is sent back to the large language model API generation unit.

[0011] An API automatic feedback method based on a large language model, the method comprising:

[0012] Generate API requests based on large language models.

[0013] The API request performs a first-level error detection in the local error type document through the static scanning component, and outputs the first feedback information. If the first feedback information is an error, the API request, instruction data, and API document are fed back to the large language model for a calibration cycle of the API request, and the real API request is obtained according to the preset hyperparameters. Conversely, the server executes the API request, instruction data API document, generates a response message, and inputs the response information into the dynamic analysis component for a second-level error detection, and outputs the second feedback information. If the second feedback information is an error, dynamic feedback information is generated based on the response information and the API document, and the dynamic feedback information is fed back to the large language model for a second-level calibration cycle of the API request, and the real API request is obtained according to the preset hyperparameters. Conversely, the real API request is output according to the response information.

[0014] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0015] Generate API requests based on large language models.

[0016] The API request performs a first-level error detection in the local error type document through the static scanning component, and outputs the first feedback information. If the first feedback information is an error, the API request, instruction data, and API document are fed back to the large language model for a calibration cycle of the API request, and the real API request is obtained according to the preset hyperparameters. Conversely, the server executes the API request, instruction data API document, generates a response message, and inputs the response information into the dynamic analysis component for a second-level error detection, and outputs the second feedback information. If the second feedback information is an error, dynamic feedback information is generated based on the response information and the API document, and the dynamic feedback information is fed back to the large language model for a second-level calibration cycle of the API request, and the real API request is obtained according to the preset hyperparameters. Conversely, the real API request is output according to the response information.

[0017] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0018] Generate API requests based on large language models.

[0019] The API request performs a first-level error detection in the local error type document through the static scanning component, and outputs the first feedback information. If the first feedback information is an error, the API request, instruction data, and API document are fed back to the large language model for a calibration cycle of the API request, and the real API request is obtained according to the preset hyperparameters. Conversely, the server executes the API request, instruction data API document, generates a response message, and inputs the response information into the dynamic analysis component for a second-level error detection, and outputs the second feedback information. If the second feedback information is an error, dynamic feedback information is generated based on the response information and the API document, and the dynamic feedback information is fed back to the large language model for a second-level calibration cycle of the API request, and the real API request is obtained according to the preset hyperparameters. Conversely, the real API request is output according to the response information.

[0020] The above-mentioned API automatic feedback system, method and device based on a large language model are mainly composed of a server, a large language model generating API request unit, a static scanning component (SSC) and a dynamic analysis component (DAC), wherein SSC and DAC respectively implement static and dynamic feedback loops for erroneous API requests generated by LLM. First, the problems existing in the prior art are summarized, and the main error types of API requests generated by LLM are obtained, and they are added to the static feedback as factual information, which effectively alleviates the problem of hallucinations caused by the lack of factual information in the traditional static feedback method. The fine-grained static feedback in the system reduces the source of errors, which is conducive to LLM accurately repairing erroneous API requests. Dynamic feedback automates the API document retrieval process, uses the error information in the API response as an index, queries the relevant error causes in the API document, and enhances the amount of information in the dynamic feedback. It effectively alleviates the problem that the traditional dynamic feedback method cannot stimulate the ability of LLM to correct errors in API requests due to insufficient information. At the same time, the two loops of static and dynamic feedback can jointly promote the generation of correct API requests, achieving performance far exceeding that of a single feedback, and significantly reducing the overhead of interacting with LLM to complete tasks. Finally, the automatic feedback framework AutoFeedback provides a human-computer assisted interface. For problems that cannot be solved through automatic feedback, users can refer to the logs generated during the feedback process and manually provide more specific feedback or other methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a structural block diagram of an API automatic feedback system based on a large language model in one embodiment;

[0022] Figure 2 1 is a flow chart of an automatic API feedback method based on a large language model in one embodiment;

[0023] Figure 3 A schematic diagram of a static feedback template for automatic API feedback based on a large language model in one embodiment;

[0024] Figure 4 A schematic diagram of a process of automatic API feedback and dynamic feedback based on a large language model in one embodiment;

[0025] Figure 5 A schematic diagram of the design process of an LLM-based API request generation framework AutoFeedback in one embodiment;

[0026] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0028] In one embodiment, Figure 1 As shown, an API automatic feedback system based on a large language model is provided, including: a server 102, a large language model API generation unit 104, a static scanning component 106 and a dynamic analysis component 108, wherein:

[0029] Server 102 acts as a proxy for the real server corresponding to the collected API, and is used to maintain a routing table of an API and the real server corresponding to the API, receive API requests and forward the API requests to the address of the real server, and integrate the execution results returned by the real server into an API response.

[0030] The large language model API generation unit 104 is used to parse the instruction information and the API document, output the API request corresponding to the instruction information, and calibrate the information content of the API request according to the received static feedback information and dynamic feedback information.

[0031] The static scanning component 106 is used to respond to an API request sent to the server for execution. When there is an error before the server executes the request, the static scanning component 106 detects a preset error type based on the API generated by the received large language model API generation unit, generates static feedback information of the API request, and transmits the static feedback information back to the large language model API generation unit.

[0032] It is worth noting that the static scanning component performs local scanning, matches the preset error type, and generates static feedback to the large model before the API request generated by the large language model is sent to the server for execution.

[0033] The dynamic analysis component 108 is used to respond to an error after an API request is sent to the server for execution. According to the API generated by the received large language model API generation unit and the execution result returned by the server, the dynamic analysis component 108 performs real-time information reasoning using the API document to generate dynamic feedback information for executing the API request, and transmits the dynamic feedback information back to the large language model API generation unit.

[0034] It is worth noting that the dynamic analysis component obtains the execution result after the API request generated by the large language model is sent to the server for execution. It then retrieves the API document to obtain the detailed error content of the execution result, forms dynamic feedback, and feeds it back to the large model.

[0035] In one of the embodiments, the preset error types include: E1-type sub-errors, E2-type sub-errors, E3-type sub-errors, and E4-type sub-errors. E1-type sub-errors include: the content generated by the large language model API generation unit cannot be parsed into the API. E2-type sub-errors include: the large language model API generation unit calls the wrong API name. E3-type sub-errors include: the large language model API generation unit calls the wrong parameter name. E4-type sub-errors include: the large language model API generation unit calls the parameter type inconsistent with the API document.

[0036] It is worth noting that if the LLM (Large Language Model) cannot generate the correct API format, it will be classified as E1. Regarding the sources of E2 API name errors, there are mainly three dimensions: E2. Selection error: calling other candidate API names; E2.2 Character error: incorrect case of letters in the API name, or different naming methods (such as camel case method, underscore method); E2.3 Semantic error: semantically similar to the correct API name. Similar to E2, the sources of E3 parameter errors also come from these three dimensions: E3.1 Selection error: calling parameter names in other candidate APIs; E3.2 Character error: incorrect case of letters in the parameter name, or different naming methods; E3.3 Semantic error: semantically similar to the correct parameter name. It is difficult for rule-based methods to locate the wrong type of parameter values, and E4.1 type error is proposed: the parameter type is inconsistent with the API document. These predefined error types will be used as pseudo-facts for feedback, which greatly reduces the illusion of LLM when generating API requests.

[0037] In one embodiment, the static scanning component 106 includes: an API retriever and an automatic error detection unit. The static scanning component is further used to receive instruction information and API documents, and the API retriever recommends an API name that meets the instruction information, so as to locate the E2 class sub-error corresponding to the API request through the automatic error detection unit to generate static feedback information.

[0038] It is worth noting that the API retriever recommends the API name in the API document that best matches the instruction information to form a recommended API subset; it only detects E2 errors, and other errors are detected by parsing API requests and comparing them with API documents. Among them, the component input is: the input of the static scanning component 108 (Static Scanning Component, SSC) includes user instructions, API documents (including API names and their descriptions, types and descriptions of required parameters, exception descriptions, etc.) and API requests in the format of APINAME (key1=value1, key2=value2, ...) generated by LLM; on this basis, the input of the dynamic analysis component (Dynamic Analysis Component, DAC) also includes API responses from the API server.

[0039] In one embodiment, the dynamic analysis component 108 includes: a document retrieval unit and a dynamic feedback unit. The dynamic analysis component is also used to receive feedback information after the API request is executed on the server. The document retrieval unit uses the retrieval augmented generation algorithm (RAG) to extract error information from the feedback information, extracts the error information and uses the embedded model for vectorization, and uses the output vector as an index to retrieve the dynamic error text database built based on the API document. And the ReAct algorithm is used to infer the factual information of the large language model API generation unit responding to the API request based on the feedback information, and the factual information is matched with the dynamic error text database to generate dynamic feedback information.

[0040] It is worth noting that the dynamic error text database is constructed according to the API documentation and API hierarchy; during the operation, the API response is input into the embedding model to form an index vector, and then the vector is used to search the dynamic error text database to find the text most relevant to the API response, expand the original API response, obtain a detailed API error description, and form dynamic feedback. Among them, DAC focuses on the fact that many APIs return insufficient information (such as error codes) after being executed on the server, rather than detailed error messages. In this case, the user must continue to query the API documentation to find a detailed explanation of the corresponding error code. This process often takes up a lot of the user's time. DAC automates this retrieval, improves the level of detail of dynamic feedback, and helps correct corresponding errors in API requests.

[0041] For the specific definition of the API automatic feedback device based on a large language model, please refer to the definition of the API automatic feedback method based on a large language model above, which will not be repeated here. Each module in the above-mentioned API automatic feedback device based on a large language model can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0042] In one embodiment, Figure 2 As shown in FIG, a method for automatic API feedback based on a large language model is provided, and the method is applied to Figure 1 The feedback system in is taken as an example, which includes the following steps:

[0043] Step 202: Generate an API request based on the large language model.

[0044] Specifically, LLM takes the user's instructions as input and generates API requests according to the API documentation.

[0045] Step 204: The API request performs a primary error detection in a local error type document through a static scanning component, and outputs a first feedback message. If the first feedback message is an error, the API request, instruction data, and API document are fed back to the large language model to perform a calibration cycle of the API request, and the real API request is obtained according to the preset hyperparameters. Otherwise, the server executes the API request, instruction data API document, generates a response message, and inputs the response message into the dynamic analysis component for a secondary error detection, and outputs a second feedback message. If the second feedback message is an error, dynamic feedback information is generated based on the response information and the API document, and the dynamic feedback information is fed back to the large language model to perform a secondary calibration cycle of the API request, and the real API request is obtained according to the preset hyperparameters. Otherwise, the real API request is output according to the response message.

[0046] There are common problems in LLM-based API request generation, and four main types of errors are summarized:

[0047] E1: NO_API: No API request was generated, or the API request format was incorrect, resulting in parsing failure;

[0048] E2: API_NAME_MISMATCH: There is a mismatch between the generated API name and the format of the API documentation or the intent of the user instruction;

[0049] E3:PARAMETER_INVALID: The generated output contains invalid parameter names that do not appear in the API documentation;

[0050] · E4: INPUT_MISMATCH: The type of the parameter value does not match the API documentation or the user instruction.

[0051] Specifically, we use a deep learning model as a subcomponent of SSC, the API retriever M, which can calculate the semantic similarity between two texts:

[0052] M(text 1 ,text 2 )=S,S∈[0,1]

[0053] The model M encodes the user instructions and API descriptions in the document into two embeddings and calculates their relevance through embedding similarity. Then, the APIs in the document are ranked according to the similarity score between the description and the instruction, and SSC lists the k APIs with the highest scores (usually k is set to 1) as the relevant API set R, where R is the API name in the subset of the API document that is considered to meet the user's needs and will be provided to the subsequent error detection subcomponent.

[0054] Further, the LLM output may not contain an API request or cannot be parsed, which can be summarized as E1. For other generated API requests, SSC performs the following automatic error detection:

[0055] API name (E2.1, E2.2, E2.3). First, SSC determines the API request r generated by LLM. 1 Is it in R, if not The API name is incorrect (E2). The next step is to identify the source of the error and refine the error type. If the API name appears in the API documentation, the sub-error type is E2.1. This is considered to be a misunderstanding of the user's intention by LLM and incorrectly selecting a different API. Due to different coding styles, LLM may have incorrectly generated a name with underscores (such as user_login), whose true value should be a camel case method name (userLogin). SSC changes all API names to lowercase and removes characters other than alphanumeric characters through a regular expression (r'[a-zA-Z]'). After regularization, if there is an APIr in the API documentation 2 With r 1 If the names are the same, the sub-error type is E2.2. The detection of E2.3 is mainly aimed at LLM to fabricate non-existent and fake APIs in hallucinations. For example, the user instruction is "I'm trying to find out how much aspirin is left." The generated fake API is "find_aspirin_number()", while the real API is "list_medicines(name='aspirin')". M is also used to detect r 1 The semantic similarity score between the name of the candidate API and the name of the candidate API in the API documentation. 3 If the score exceeds the threshold (usually set to 0.5), the sub-error type is E2.3. Other cases are "other" errors in E2.

[0056] API parameter name (E3.1, E3.2, E3.3). 1 ∈R, SSC next verifies the correctness of the API parameter name. Unlike the API name, the correct parameter name is already provided in the API documentation. 1 If the parameter name in the API document does not exist in the corresponding description, it is considered as E3. The incorrect API parameter is considered as p 1 If p appears in the parameter list of other APIs in the documentation 1 The sub-error type is E3.1. The SSC operation on the API parameter name is the same as the operation on the API name in E2.2 and E2.3. 1 Same as parameter p in other APIs 2 When matching, the sub-error type is E3.2. If the parameter p in the API documentation 3 With p 1 If the similarity score of the name is greater than the threshold, it is considered to be E3.3. Otherwise, it is an "other" error in E3.

[0057] API parameter value (E4.1). Due to the great flexibility of parameter values, only a preliminary judgment can be made based on the parameter type. SSC analysis 1 and compares them with the data types specified in the API documentation. 1 Matching error, the corresponding parameter is p v In some cases, there is no error after static scanning, but there is an error during the call process, or the call result still cannot meet the user's requirements, which is the "other" error in E4.

[0058] Return value. SSC detects errors in ascending numerical order (such as E1→E2, E2.2→E2.3). Once an error is detected, it returns the corresponding value to enter the next step of static feedback. In addition to returning the error type, other return values ​​are shown in Table 1.

[0059] Table 1 Return values ​​of the error detection phase

[0060]

[0061] 3) Static Feedback: SSC receives the generated API request as input, performs error detection with the help of API retriever, and generates static errors. Then SSC selects the corresponding prompt template according to the error type and fills it in the given format according to other return values ​​to generate static feedback. Figure 3 As shown, the template is roughly divided into five parts, and the granularity of feedback gradually becomes finer. First, declare that there is an error in the API request; second, inform the location and content of the error; third, eliminate the source of the error that has passed the detection (for E2.3, E2.1 and E2.2 are excluded); fourth, find the source of the error and provide correction suggestions; fifth, request LLM to regenerate the API request. Some static feedback of errors may miss some parts. For example, in E1, since only one detection was performed, there is no third part in the feedback.

[0062] Furthermore, DAC retrieves detailed information of the corresponding error from the API document. The Retrieval Augmented Generation (RAG) method is used to extract information related to the error. First, the API document is split to obtain all document contents related to the API (including function descriptions, exception instructions, etc.). Then, these contents are split into sentence granularity and the sentences are vectorized using an embedding model. Next, semantically similar sentences are put into the same data block to achieve semantic chunking and vectorization. This is equivalent to building a vector-indexed text database with vectorized sentences as indexes.

[0063] Further, in Figure 4 In the retrieval process shown, DAC uses the API name (such as route_planing) to match the corresponding block vector. Then the same embedding model is used to vectorize the user query and find the most similar block vector as an index (implemented by the approximate nearest neighbor algorithm) to retrieve the corresponding text from the previously built database. Finally, more detailed error information is obtained.

[0064] Furthermore, the ReAct algorithm is integrated into DAC, allowing LLM to observe API responses and incorporate factual information into reasoning. At the same time, the error message from the document retrieval subcomponent is combined with the API response to form dynamic feedback. The specific steps are as follows:

[0065] The server executes the sent API request and obtains the corresponding API response. The next step is to evaluate whether the user's needs are met or whether the maximum feedback count is reached. If the conditions are met, the task ends and exits. Otherwise, it enters a dynamic feedback loop: DAC first retrieves the corresponding error message in the document; the ReAct algorithm takes the API request as an action, combines the API response and the error message as an observation, and feeds it back to the LLM together with the historical interaction record; the newly generated API request and thought will be stored as a feedback record together with the action and observation. Finally, the new execution result returned by the server is evaluated.

[0066] In the above-mentioned API automatic feedback method based on a large language model, the API automatic feedback system applied to the large language model realizes the static and dynamic feedback loops for the erroneous API requests generated by the LLM respectively. First, the problems existing in the prior art are summarized, and the main error types of API requests generated by the LLM are obtained, and they are added to the static feedback as factual information, which effectively alleviates the problem of hallucinations caused by the lack of factual information in the traditional static feedback method. In addition, the fine-grained static feedback in the system narrows the source of errors, which is conducive to the accurate repair of erroneous API requests by the LLM. Dynamic feedback automates the API document retrieval process, uses the error information in the API response as an index, and queries the relevant error causes in the API document, thereby enhancing the amount of information in the dynamic feedback. It effectively alleviates the problem that the traditional dynamic feedback method cannot stimulate the ability of the LLM to correct the errors of the API request due to insufficient information. At the same time, the two loops of static and dynamic feedback can jointly promote the generation of correct API requests, achieving performance far exceeding that of a single feedback, and significantly reducing the overhead of interacting with the LLM to complete the task. Finally, the AutoFeedback framework provides a human-computer assisted interface. For problems that cannot be solved by automatic feedback, users can refer to the logs generated during the feedback process and manually provide more specific feedback or other methods.

[0067] In one of the embodiments, the instruction data input by the user is parsed and matched with the local API document of the server based on the large language model to generate an API request corresponding to the instruction data.

[0068] In one embodiment, if the first feedback information is an error, the semantic similarity between the instruction data and the API document is calculated by the deep learning model of the static scanning component, and the API subset to be recommended is obtained according to the semantic similarity. The API subset to be recommended is retrieved by the error detection unit of the static scanning component for local error type, and the error API request after the first level of feedback and the instruction data corresponding to the error API request are output to the large language model for calibration loop, and the real API request is obtained according to the preset hyperparameters as loop constraints.

[0069] It is worth noting that the subset of recommended APIs in the static scanning component is only used to detect E2 type errors, and the remaining error types are judged by comparing the parsed API with the API documentation. LLM takes the API documentation and user instructions as input and parses the output it generates to obtain the API request. SSC performs local static scanning to determine whether there are any errors in the API request. If not, continue to send the request; if so, error detection is performed to locate the sub-error and provide accurate static feedback to LLM.

[0070] In one of the embodiments, the response information includes: an API request, a response result of executing the API request, and instruction data corresponding to the API request. If the second feedback information is an error, the response error information is extracted and vectorized using an embedding model through the retrieval enhancement generation algorithm adopted by the document retrieval unit of the dynamic analysis component. The dynamic error text database built based on the API document is retrieved with the output vector as the index to obtain the API detailed error information. The response fact information of the API request is inferred based on the API detailed error information and the ReAct algorithm to generate dynamic feedback information, and the dynamic feedback information is fed back to the large language model for a secondary calibration loop of the API request. The real API request is obtained based on the preset hyperparameters as loop constraints.

[0071] It is worth noting that LLM uses its own internal representation to produce output that is not based on the external world, and it is crucial for tools to enhance the interaction of LLM with external tools, which can mitigate the possibility of fact hallucinations.

[0072] In one embodiment, if Figure 5 As shown in the figure, a design flow of an LLM-based API request generation framework AutoFeedback is provided. The automatic feedback in the AutoFeedback framework includes two low-coupled complementary components: a static scanning component (SSC) and a dynamic analysis component (DAC). They are combined with the original process of LLM calling the API (gray arrows) to create two feedback loops. In order to avoid an infinite loop, the maximum number of feedbacks is set and used as a hyperparameter.

[0073] 1) Static feedback flow (dashed arrow). LLM takes API documents and user instructions as input and parses the output it generates to obtain API requests. SSC performs local static scanning to determine whether there are any errors in the API request. If not, it continues to send the request; if so, it performs error detection to locate the sub-errors and provide precise static feedback to LLM.

[0074] 2) Dynamic feedback process (dotted arrows). After the API request is sent to the server and executed, the user evaluates whether the returned API response meets their needs. For those failed API requests, DAC retrieves their detailed error information in the API documentation and combines it with the API response as dynamic feedback.

[0075] It is worth noting that SSC is mainly oriented towards helping generate API requests that follow user instructions and format standards. It is also worth mentioning that SSC cannot feedback details about parameter values. However, this does not mean that SSC is not beneficial for correcting parameter values, as it narrows down the source of errors and allows LLM in static and dynamic feedback to focus on parameter values.

[0076] It should be understood that although Figure 2 , Figure 5 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 , Figure 5 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0077] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an API automatic feedback method based on a large language model is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0078] Those skilled in the art will understand that Figure 1 , Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0079] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0080] Generate API requests based on large language models.

[0081] The API request performs a first-level error detection in the local error type document through the static scanning component, and outputs the first feedback information. If the first feedback information is an error, the API request, instruction data, and API document are fed back to the large language model for a calibration cycle of the API request, and the real API request is obtained according to the preset hyperparameters. Conversely, the server executes the API request, instruction data API document, generates a response message, and inputs the response information into the dynamic analysis component for a second-level error detection, and outputs the second feedback information. If the second feedback information is an error, dynamic feedback information is generated based on the response information and the API document, and the dynamic feedback information is fed back to the large language model for a second-level calibration cycle of the API request, and the real API request is obtained according to the preset hyperparameters. Conversely, the real API request is output according to the response information.

[0082] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0083] Generate API requests based on large language models.

[0084] The API request performs a first-level error detection in the local error type document through the static scanning component, and outputs the first feedback information. If the first feedback information is an error, the API request, instruction data, and API document are fed back to the large language model for a calibration cycle of the API request, and the real API request is obtained according to the preset hyperparameters. Conversely, the server executes the API request, instruction data API document, generates a response message, and inputs the response information into the dynamic analysis component for a second-level error detection, and outputs the second feedback information. If the second feedback information is an error, dynamic feedback information is generated based on the response information and the API document, and the dynamic feedback information is fed back to the large language model for a second-level calibration cycle of the API request, and the real API request is obtained according to the preset hyperparameters. Conversely, the real API request is output according to the response information.

[0085] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0086] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0087] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. An API automatic feedback system based on a large language model, characterized in that: The system includes: a server, a large language model API generation unit, a static scanning component and a dynamic analysis component; The server acts as a proxy for the real servers corresponding to the collected APIs, and is used to maintain a routing table of the APIs and the real servers corresponding to the APIs, receive API requests and forward the API requests to the addresses of the real servers, and integrate the execution results returned by the real servers into API responses; The large language model API generation unit is used to parse the instruction information and the API document, output the API request corresponding to the instruction information, and calibrate the information content of the API request according to the received static feedback information and dynamic feedback information; The static scanning component is used to detect a preset error type and generate static feedback information of the API request according to the received API generated by the large language model API generation unit when there is an error before the API request is sent to the server for execution, and transmit the static feedback information back to the large language model API generation unit; The dynamic analysis component is used to respond to the API request and, when an error occurs after it is sent to the server for execution, perform real-time information reasoning using the API document based on the API generated by the received large language model API generation unit and the execution result returned by the server, generate dynamic feedback information for executing the API request, and transmit the dynamic feedback information back to the large language model API generation unit.

2. The system according to claim 1, characterized in that The preset error types include: E1 sub-error, E2 sub-error, E3 sub-error, and E4 sub-error; The E1-type sub-errors include: the content generated by the large language model API generation unit cannot be parsed into the API; The E2-type sub-errors include: the large language model API generation unit calls an incorrect API name; The E3 type sub-errors include: the large language model API generation unit calls an incorrect parameter name; The E4 type sub-error includes: the large language model API generation unit call parameter type is inconsistent with the API document.

3. The system according to claim 2, characterized in that The static scanning component includes: an API retriever and an automatic error detection unit; The static scanning component is also used to receive the instruction information and the API document, and the API retriever recommends an API name that conforms to the instruction information, so as to locate the E2 class sub-error corresponding to the API request through the automatic error detection unit to generate static feedback information.

4. The system according to claim 3, characterized in that The dynamic analysis component includes: a document retrieval unit and a dynamic feedback unit; The dynamic analysis component is also used to receive feedback information after the API request is executed on the server, the document retrieval unit uses a retrieval enhancement generation algorithm to extract error information from the feedback information, extracts the error information and uses an embedded model to vectorize it, and uses the output vector as an index to retrieve a dynamic error text database built based on the API document; and uses the ReAct algorithm to infer fact information of the large language model API generation unit in response to the API request based on the feedback information, matches the fact information with the dynamic error text database, and generates dynamic feedback information.

5. An API automatic feedback method based on a large language model, characterized in that: The method comprises: Generate API requests based on large language models; The API request performs a primary error detection in a local error type document through a static scanning component, and outputs a first feedback information. If the first feedback information is an error, the API request, instruction data, and API document are fed back to the large language model to perform a calibration cycle of the API request, and a real API request is obtained according to preset hyperparameters; otherwise, the server executes the API request, instruction data, and API document, generates response information, and inputs the response information into a dynamic analysis component to perform a secondary error detection, and outputs a second feedback information; if the second feedback information is an error, dynamic feedback information is generated according to the response information and the API document, and the dynamic feedback information is fed back to the large language model to perform a secondary calibration cycle of the API request, and a real API request is obtained according to the preset hyperparameters; otherwise, the real API request is output according to the response information.

6. The method according to claim 5, characterized in that Generate API requests based on large language models, including: The command data input by the user is parsed and matched with the local API document of the server based on the large language model to generate an API request corresponding to the command data.

7. The method according to claim 6, characterized in that If the first feedback information is an error, the API request, instruction data, and API document are fed back to the large language model to perform a calibration cycle of the API request, and a real API request is obtained according to preset hyperparameters, including: If the first feedback information is an error, the semantic similarity between the instruction data and the API document is calculated through the deep learning model of the static scanning component, and the API subset to be recommended is obtained according to the semantic similarity; the API subset to be recommended is retrieved by the error detection unit of the static scanning component through local error type retrieval, and the error API request after the first-level feedback and the instruction data corresponding to the error API request are output to the large language model for calibration loop, and the real API request is obtained according to the preset hyperparameters as loop constraints.

8. The method according to claim 7, characterized in that The response information includes: the API request, the response result of executing the API request, and the instruction data corresponding to the API request; If the second feedback information is an error, dynamic feedback information is generated according to the response information and the API document, and the dynamic feedback information is fed back to the large language model to perform a secondary calibration cycle of the API request, and a real API request is obtained according to preset hyperparameters, including: If the second feedback information is an error, the retrieval enhancement generation algorithm adopted by the document retrieval unit of the dynamic analysis component is used to extract the response error information and vectorize it using the embedding model. The dynamic error text database built based on the API document is retrieved with the output vector as the index to obtain API detailed error information. The response fact information of the API request is inferred based on the API detailed error information and the ReAct algorithm to generate dynamic feedback information. The dynamic feedback information is fed back to the large language model to perform a secondary calibration loop of the API request, and the real API request is obtained based on the preset hyperparameters as loop constraints.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 5 to 8 are implemented.

Images