Intelligent agent model self-adaptive optimization method and system based on error feedback information

Abstract

The application relates to the technical field of artificial intelligence, and discloses an intelligent agent model self-adaptive optimization method and system based on error feedback information, which are used for solving the technical problem that an intelligent agent model cannot automatically execute optimization according to user feedback information, and the accuracy of output answers and the degree of automation are low in the prior art. The method comprises the following steps: receiving error feedback information of a user from a feedback interface; obtaining a verifier template corresponding to the error feedback information, and generating a feedback verifier based on the content of the error feedback information and the verifier template, and storing the feedback verifier in a dynamic verifier library; when a background optimization process is triggered, obtaining a simulated questioning instruction, obtaining a candidate response generated by an intelligent agent model based on the simulated questioning instruction, searching for a candidate feedback verifier related to the simulated questioning instruction in the dynamic verifier library, calling each candidate feedback verifier to calculate a reward signal, updating policy parameters of the intelligent agent model based on the reward signal, and optimizing the intelligent agent model.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an adaptive optimization method, system, electronic device, computer storage medium, and computer program product for an intelligent agent model based on error feedback information. Background Technology

[0002] With the development of artificial intelligence, intelligent agent question-answering systems, built around large language models and based on Retrieval-Augmented Generation (RAG) technology, have been widely applied in scenarios such as intelligent customer service and intelligent assistants. However, since these intelligent agent models primarily rely on manually labeled datasets or manually constructed information graphs to provide answers, they are prone to failing to provide correct answers when information in the database or graph is insufficient. Furthermore, due to the characteristics of large language models, they may suffer from the "model illusion" problem, where generated content does not match reality. These issues all contribute to the inaccuracy of the results output by intelligent agent models.

[0003] To address the issue of inaccurate output from intelligent agent models, several optimization schemes exist in existing technologies. For example, some methods train the model using high-quality, large-scale datasets, periodically collect feedback, and optimize and update the dataset based on the feedback results. However, this training process is offline, requiring periodic shutdowns of the intelligent agent model, and incurs high labor and time costs. Alternatively, automated reinforcement learning schemes can be employed, training the reward model by collecting user preferences for multiple responses to optimize itself. However, this optimization approach is based on subjective user preferences rather than facts, resulting in poor optimization performance.

[0004] Therefore, there is an urgent need in this field for a method that can adaptively and automatically optimize the model based on feedback information, automatically generate a validator based on error feedback information provided by the model user in real time, automatically perform background optimization based on the validator, and automatically optimize the intelligent agent model to improve the accuracy and automation of the intelligent agent model's answers. Summary of the Invention

[0005] The main objective of this invention is to solve the technical problem that existing intelligent agent models cannot automatically perform optimization based on user feedback, resulting in low accuracy and automation of the output answers.

[0006] The first aspect of this invention provides an adaptive optimization method for an intelligent agent model based on error feedback information, comprising: receiving error feedback information from a user on the output content of the intelligent agent model from a feedback interface;

[0007] Obtain the validator template corresponding to the error feedback information, generate a feedback validator based on the content of the error feedback information and the validator template, and store it in the dynamic validator library;

[0008] When the background optimization process is triggered, a simulated question instruction is obtained, and the candidate response generated by the intelligent agent model based on the simulated question instruction is obtained. The candidate feedback validator related to the simulated question instruction is searched in the dynamic validator library, and each candidate feedback validator is called to calculate the reward signal based on the candidate response. The strategy parameters of the intelligent agent model are updated based on the reward signal to optimize the intelligent agent model.

[0009] Optionally, in a first implementation of the first aspect of the present invention, obtaining the verifier template corresponding to the error feedback information includes:

[0010] The error feedback information is semantically parsed and intent identified to determine the correction intent type of the current error feedback information;

[0011] Based on the aforementioned correction intent type, the corresponding validator template is searched in the preset template library.

[0012] Optionally, in a second implementation of the first aspect of the present invention, the type of correction intent includes fact correction, calculation error correction, and format error correction.

[0013] Optionally, in a third implementation of the first aspect of the present invention, the step of searching for candidate feedback validators related to the simulated question instruction in the dynamic validator library and calling each candidate feedback validator to calculate the reward signal based on the candidate response includes:

[0014] Based on semantic similarity, candidate feedback validators related to the question instruction are found and simulated;

[0015] Based on the candidate feedback validator, the simulated question instruction is verified and rewarded to obtain the reward output value of each candidate feedback validator;

[0016] The reward signal is obtained by weighted summation of the reward output values ​​of each candidate feedback validator.

[0017] Optionally, in a fourth implementation of the first aspect of the present invention, after generating the feedback validator based on the content of the error feedback information and the validator template, the method further includes:

[0018] Perform feasibility testing on the generated feedback validator;

[0019] If the feedback validator passes the feasibility test, then the feedback validator is retained;

[0020] If the feedback validator fails the feasibility test, then the feedback validator is deleted.

[0021] Optionally, in a fifth implementation of the first aspect of the present invention, after receiving error feedback information from the feedback interface regarding the user's output of the intelligent agent model, the method further includes:

[0022] The intelligent agent model is invoked to perform semantic analysis on the error feedback information to determine whether there is any ambiguity.

[0023] If there is any ambiguity, a clarifying question will be sent to the user based on the error feedback information, and the user's clarifying answer will be accepted.

[0024] Based on the clarification response, the ambiguous content of the error feedback information was corrected.

[0025] A second aspect of the present invention provides an adaptive optimization system for an intelligent agent model based on error feedback information, comprising:

[0026] The feedback receiving module is used to receive error feedback information from the user regarding the output content of the intelligent agent model from the feedback interface;

[0027] The verifier generation module is used to obtain the verifier template corresponding to the error feedback information, generate a feedback verifier based on the content of the error feedback information and the verifier template, and store it in the dynamic verifier library.

[0028] The optimization execution module is used to obtain a simulated question instruction when the background optimization process is triggered, and obtain the candidate response generated by the intelligent agent model based on the simulated question instruction. It searches for candidate feedback validators related to the simulated question instruction in the dynamic validator library, calls each candidate feedback validator to calculate the reward signal based on the candidate response, updates the strategy parameters of the intelligent agent model based on the reward signal, and optimizes the intelligent agent model.

[0029] A third aspect of the present invention provides an intelligent agent model adaptive optimization device based on error feedback information, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor invokes the instructions in the memory to cause the intelligent agent model adaptive optimization device based on error feedback information to perform the steps of the above-described intelligent agent model adaptive optimization method based on error feedback information.

[0030] A fourth aspect of the present invention provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the steps of the above-described adaptive optimization method for an intelligent agent model based on error feedback information.

[0031] A fifth aspect of the present invention provides a computer program product comprising a computer program / instruction that, when executed by a processor, implements the steps of the intelligent agent model adaptive optimization method based on error feedback information as described above.

[0032] The technical solution provided by this invention receives error feedback information from the user regarding the output content of the intelligent agent model from a feedback interface; obtains the validator template corresponding to the error feedback information, generates a feedback validator based on the content of the error feedback information and the validator template, and stores it in a dynamic validator library; when the background optimization process is triggered, obtains a simulated question instruction and the candidate response generated by the intelligent agent model based on the simulated question instruction, searches for candidate feedback validators related to the simulated question instruction in the dynamic validator library, calls each candidate feedback validator to calculate the reward signal based on the candidate response, updates the strategy parameters of the intelligent agent model based on the reward signal, and optimizes the intelligent agent model. This method can automatically optimize the intelligent agent model based on the error feedback information provided by the user, adaptively improve the output accuracy of the intelligent agent model, and enhance the automation level of the intelligent agent model.

[0033] Furthermore, the system, electronic device, computer-readable storage medium, and computer program product provided by this invention also solve the corresponding technical problems. Attached Figure Description

[0034] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0035] Figure 1 This is a flowchart illustrating the first embodiment of the intelligent agent model adaptive optimization method based on error feedback information in this invention.

[0036] Figure 2 This is a flowchart illustrating the second embodiment of the intelligent agent model adaptive optimization method based on error feedback information in this invention.

[0037] Figure 3 This is a schematic diagram of an embodiment of the intelligent agent model adaptive optimization system based on error feedback information in this invention.

[0038] Figure 4 This is a schematic diagram of an embodiment of the intelligent agent model adaptive optimization device based on error feedback information in this invention.

[0039] Figure 5 This is a schematic diagram illustrating the principle of a computer-readable medium according to an embodiment of the present invention. Detailed Implementation

[0040] Exemplary embodiments of the invention will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limiting the invention to the embodiments set forth herein. Rather, these exemplary embodiments are provided to make the invention more comprehensive and complete, and to facilitate a full communication of the inventive concept to those skilled in the art. The same reference numerals in the drawings denote the same or similar elements, components, or parts, and therefore repeated descriptions of them will be omitted.

[0041] Subject to the technical concept of this invention, the features, structures, characteristics or other details described in a particular embodiment may be combined in one or more other embodiments in a suitable manner.

[0042] In the description of specific embodiments, the features, structures, characteristics, or other details described in this invention are intended to enable those skilled in the art to fully understand the embodiments. However, it is not excluded that those skilled in the art can practice the technical solutions of this invention without one or more of the specific features, structures, characteristics, or other details.

[0043] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0044] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0045] The terms “and / or” or “and / or” include all combinations of any one or more of the listed items.

[0046] Please see Figure 1 The first embodiment of the intelligent agent model adaptive optimization method based on error feedback information in this invention includes:

[0047] S101. Receive error feedback information from the user regarding the output content of the intelligent agent model from the feedback interface;

[0048] It is understood that the executing entity of this invention can be an intelligent agent model adaptive optimization device or system based on error feedback information, or it can be a terminal or a server; the specific implementation is not limited here. This embodiment of the invention will be described using a server as an example.

[0049] In this embodiment, users can interact with an Intelligent Agent model deployed on a server via a user terminal. Upon receiving a question from the user, the Intelligent Agent model generates an intelligent answer based on the semantic content of the question and sends it to the user as its output. If the user finds an error in the output and needs to provide feedback, they can send error feedback information to the Intelligent Agent model through a feedback interface. Subsequently, the Intelligent Agent model can receive the user's error feedback information regarding its output from the feedback interface.

[0050] S102. Obtain the validator template corresponding to the error feedback information, generate a feedback validator based on the content of the error feedback information and the validator template, and store it in the dynamic validator library.

[0051] Upon receiving error feedback information, semantic parsing and intent recognition are performed on the error feedback information to determine the correction intent type of the current error feedback information. The correction intent type includes fact correction, calculation error correction, and format error correction. Subsequently, based on the correction intent type, the corresponding validator template is searched in a preset template library.

[0052] The preset template library described in this embodiment is a predefined, modular collection of function templates stored in the system's database or constants. The validator template is a parameterizable function skeleton containing placeholders. The parameter information from the error feedback information is dynamically injected into the function skeleton according to the placeholder markers to generate a reward function. Based on this reward function, a feedback validator is generated.

[0053] In addition, after generating the feedback validator, a feasibility test is performed on the generated feedback validator; if the feedback validator passes the feasibility test, the feedback validator is retained and stored in the dynamic validator library; if the feedback validator fails the feasibility test, the feedback validator is deleted.

[0054] S103. When the background optimization process is triggered, obtain the simulated question instruction and the candidate response generated by the intelligent agent model based on the simulated question instruction. Search for the candidate feedback validator related to the simulated question instruction in the dynamic validator library, and call each candidate feedback validator to calculate the reward signal based on the candidate response. Update the strategy parameters of the intelligent agent model based on the reward signal to optimize the intelligent agent model.

[0055] In this embodiment, the server runs an asynchronous reinforcement learning loop in the background. When the background optimization process of this asynchronous reinforcement learning loop is triggered, the server starts the reinforcement learning optimizer. This optimizer can obtain server problem logs or a dedicated training dataset, find the problem domain and related content to be optimized, and generate multiple simulated question commands in the background. Based on these simulated question commands, it generates multiple corresponding answers to obtain candidate responses. After obtaining candidate responses, based on semantic similarity, it finds candidate feedback validators related to the simulated question commands, and performs verification and reward calculation on the simulated question commands based on the candidate feedback validators to obtain the reward output value of each candidate feedback validator. The reward output values ​​of each candidate feedback validator are weighted and summed to obtain the reward signal.

[0056] After receiving the reward signal, the strategy parameters of the intelligent agent model are updated based on the reward signal to optimize the intelligent agent model.

[0057] The method provided in this embodiment can automatically optimize the intelligent agent model based on the error feedback information provided by the user, thereby achieving the technical effect of adaptively improving the output accuracy of the intelligent agent model and enhancing the automation level of the intelligent agent model.

[0058] Please see Figure 2 The second embodiment of the intelligent agent model adaptive optimization method based on error feedback information in this invention includes:

[0059] S201. Receive the user's question request, parse the user's query and output an intelligent answer;

[0060] It is understood that the executing entity of this invention can be an intelligent agent model adaptive optimization device or system based on error feedback information, or it can be a terminal or a server; the specific implementation is not limited here. This embodiment of the invention will be described using a server as an example.

[0061] In this embodiment, users can interact with an Intelligent Agent model deployed on a server via a user terminal. This Intelligent Agent model can be an intelligent customer service model, an intelligent question-and-answer model, or other models capable of outputting answers based on semantic understanding. Upon receiving a question request sent by the user through the user terminal, the Intelligent Agent model parses the user query information contained in the question request, then outputs an initial response, providing an intelligent answer, and sends the intelligent answer to the user terminal. The intelligent answer can be an answer output based on the semantics of the user query, using Retrieval-augmented Generation (RAG) technology, combined with content from a knowledge base.

[0062] S202. When a user reports an error in the intelligent response, receive the user's error feedback information regarding the output of the intelligent agent model from the feedback interface.

[0063] After receiving an intelligent response, users can provide feedback. For example, if the intelligent response is incorrect, the user can send error feedback information to the server through the feedback interface, at which point the server receives the error feedback information from the feedback interface.

[0064] In one specific implementation, since the error feedback information received in this embodiment can be in natural language, it may be ambiguous due to different users' different language expressions. After receiving the error feedback information, the intelligent agent model first calls the semantic analysis algorithm to perform semantic analysis on the error feedback information to determine whether the error feedback information is ambiguous. If ambiguity exists, it sends a clarifying question to the user based on the error feedback information and accepts the user's clarifying answer. Based on the clarifying answer, it corrects the ambiguous content of the error feedback information. For example, when an intelligent answer contains multiple numbers, the error feedback information given by the user through the feedback interface is "Incorrect number, should be an integer," but it is not possible to determine which number is the integer. At this time, the intelligent agent model will initiate a clarifying dialogue, for example, asking: "Do you mean that all numbers must be integers?" After the user answers, the ambiguous content of the error feedback information is corrected to improve the accuracy of the logic of the feedback validator generated in subsequent steps.

[0065] S203. Perform semantic parsing and intent recognition on the error feedback information to determine the correction intent type of the current error feedback information;

[0066] After obtaining accurate error feedback information, semantic analysis and intent recognition are performed on the error feedback information to determine its meaning and the type of correction intent. The correction intent types include factual correction, calculation error correction, and format error correction. Specifically, factual correction refers to the correction required when the answer provided by the intelligent agent model does not conform to objective facts; calculation error correction refers to the correction required when the intelligent agent model performs mathematical operations in its answer and obtains an incorrect result; and format error correction refers to the correction required when the intelligent agent model does not follow the user-specified or preset output format in its answer.

[0067] In this step, DIET (Dual Intent and Entity Transformer, a novel multi-task algorithm for intent classification and entity recognition) can be used to extract entity information from error feedback information while obtaining the corrected intent type.

[0068] Here's a concrete example: When a user provides the error message, "Incorrect, the founder of Company A is Zhang, not Li," the server can use a semantic recognition algorithm to identify this feedback as a fact correction. When a user provides the error message, "The total price is wrong; it should be 80 yuan," the server can use a semantic recognition algorithm to identify this feedback as a calculation logic error. When a user provides the error message, "The output does not conform to the required JSON format," the server can use a semantic recognition algorithm to identify this as a structural or formatting violation.

[0069] Taking the error feedback message "No, the founder of Company A is Zhang, not Li" as an example, while using DIET to obtain the correction intent type as fact correction, the extracted content includes: entity "Company A", attribute "founder", error value "Li" and correct value "Zhang".

[0070] S204. Based on the correction intent type, find the corresponding validator template in the preset template library, and generate a feedback validator based on the content of the error feedback information and the validator template.

[0071] Based on the identified correction intent type, a corresponding validator template is searched in a pre-defined template library. This library is a predefined, modular collection of function templates, which can be stored in the system's database or constants. In this embodiment, the validator template is a parameterizable function skeleton containing clear placeholders, supporting the dynamic injection of parameters (such as entities and values) extracted from user feedback. Based on the obtained correction intent type and the extracted entities, the corresponding validator template is matched, and parameters are injected into the validator template according to information extracted from the error feedback information, generating a specific feedback validator. This feedback validator can be a specific reward function.

[0072] The following are some specific examples to illustrate this:

[0073] (1) Fact revision:

[0074] Taking the error feedback message "No, the founder of Company A is Zhang, not Li" as an example, based on the correction intent type being fact correction, after finding the validator template in the preset template library, a specific example of the generated feedback validator is as follows:

[0075] "def check_{subject}_{attribute}(response_text: str) ->float:

[0076] if "{subject}" in response_text and "{attribute}" in response_text:

[0077] correct_present = "{correct_value}" in response_text

[0078] incorrect_present = "{incorrect_value}" in response_text

[0079] if correct_present and not incorrect_present:

[0080] return 1.0

[0081] else:

[0082] return 0.0”

[0083] The specific processing flow of the above feedback validator is as follows: check whether the response text contains a subject and an attribute to ensure the relevance of the validation; when validating whether the subject's attribute in the response is correct, check whether the correct value exists and whether the known incorrect value does not exist; if the fact is correct, give a positive reward (e.g., give a reward value of 1.0); otherwise, do not give a reward (e.g., give a reward value of 0.0).

[0084] (2) Correction of calculation errors:

[0085] Taking the error feedback message "The total price is wrong, it should be 80 yuan" as an example, after finding the validator template in the preset template library, a specific example of the generated feedback validator is as follows:

[0086] “def check_calculation_{operation_name}(response_text: str) ->float:

[0087] match = re.search(r"{result_pattern}", response_text)

[0088] if match:

[0089] extracted_result = float(match.group(1))

[0090] expected = float("{expected_result}")

[0091] if abs(extracted_result - expected)<1e-6:

[0092] return 1.0

[0093] else:

[0094] return 0.0”

[0095] The specific processing flow of the above feedback validator is as follows: verify whether the calculation result in the response is the expected result (expected_result). Specifically, a regular expression (esult_pattern) is used to extract the numerical result from the response text, and the extracted result is compared with the expected result (expected_result). Considering the issue of floating-point precision, if the extracted result is basically consistent with the expected result, the calculation is correct, and a positive reward is given (e.g., a reward value of 1.0 is given); in other cases, no reward is given (e.g., a reward value of 0.0 is given).

[0096] (3) Correction of formatting errors:

[0097] Taking the error feedback message "The JSON was not output in the required format" as an example, after finding the validator template in the preset template library, a specific example of the generated feedback validator is as follows:

[0098] ""def check_format_{format_name}(response_text: str) ->float:

[0099] pattern = re.compile(r"{regex_pattern}", re.DOTALL)

[0100] if pattern.search(response_text):

[0101] return 1.0

[0102] else:

[0103] return 0.0”

[0104] The specific processing flow of the above feedback validator is as follows: Verify whether the response meets the format requirements (format_name). Obtain the regular expression describing the target format (regex_pattern), search for parts that match the regular expression. If a format that matches the regular expression is found, the format is correct, and a positive reward is given (e.g., a reward value of 1.0 is given); otherwise, no reward is given (e.g., a reward value of 0.0 is given).

[0105] In one specific implementation, if ambiguity is found in the feedback information during steps S203 and S204 (e.g., when the DIET algorithm cannot determine the user's intent or entity with high confidence), a clarifying question can be posed to the user. However, this question is not a simple inquiry, but rather a pre-generated corrective hypothesis based on the intelligent agent model's thinking ability, the context of the current erroneous feedback information, and existing knowledge. Taking the erroneous feedback information "The membership price is incorrect" as an example, the intelligent agent model would provide the following hypothesis:

[0106] Assume A: "All members enjoy a 10% discount";

[0107] Assume B: "The price for 'premium membership' should be 50 yuan."

[0108] In addition, this embodiment "virtually" generates a corresponding feedback validator code in memory for each hypothesis. This allows users to intuitively see how each choice will be translated into a machine-executable rule.

[0109] For example, feedback validator A:

[0110] "check_price_discount(response_text, user_level) ->if user_level == 'member': return response_price == original_price * 0.9"

[0111] The above feedback validator A means "a 10% discount for all members";

[0112] Feedback Validator B:

[0113] "check_price_specific(response_text, user_level) ->if user_level == 'premium': return response_price == 50.0"

[0114] The above feedback validator B means "set the exclusive price for premium members to 50 yuan".

[0115] Furthermore, the server will present the user with a structured choice, rather than an open-ended question, for example:

[0116] "Thank you for your feedback! To help me learn accurately, could you please tell me which situation you are referring to?"

[0117] Rule 1: 10% discount for all members;

[0118] Rule 2: Set the exclusive price for premium members to 50 yuan;

[0119] None of the above. I would like to provide more information.

[0120] Users can choose from the above options. When a user makes a selection, the system will officially instantiate the corresponding "virtual" validator and obtain the feedback validator.

[0121] S205. Perform a feasibility test on the generated feedback validator, retain the feedback validators that pass the feasibility test, and store them in the dynamic validator library.

[0122] Next, a feasibility test is performed on each generated feedback validator. If the feedback validator passes the feasibility test, it is retained and stored in the dynamic validator library; if the feedback validator fails the feasibility test, it is deleted and not retained.

[0123] In one specific implementation, the feasibility test described in this embodiment includes syntax checking and sandbox verification. Only if both syntax checking and sandbox verification pass will the feedback validator be stored in the dynamic validator library.

[0124] Specifically, syntax checking can be performed using Python's built-in AST (Abstract Syntax Tree) module. The generated feedback validator code (essentially a string) is passed to the AST module's `parse` function. This function attempts to parse the code string into an abstract syntax tree. If the code is syntactically correct, the `parse` function executes successfully and returns an AST object, indicating a successful check. If the code contains any syntax errors (e.g., mismatched brackets, indentation errors, invalid keywords, etc.), the `parse` function immediately throws a `SyntaxError` exception. The server catches this exception, deems the feedback validator generation a failure, and discards it.

[0125] Specifically, the sandbox validation process involved two steps: static analysis and dynamic execution.

[0126] (1) Static Analysis:

[0127] Before the code is executed, the system traverses the AST object successfully generated in the previous step, checking for any dangerous operations that have been blacklisted. For example, it checks whether the tree contains nodes related to file operations (such as the open function), network requests (such as import socket), or system command execution (such as import os).

[0128] (2) Dynamic Analysis in Sandbox:

[0129] The code that passes static analysis is executed in a highly isolated and permission-restricted sandbox environment. This is achieved using container technologies such as Docker. The environment is configured according to the "principle of least privilege," including:

[0130] (2.1) No network access: Blocks any outbound or inbound network connection;

[0131] (2.2) Read-only file system: Prohibits any write operations on the file system;

[0132] (2.3) Strict system call filtering: Using techniques such as seccomp, only a small number of absolutely safe low-level system calls are allowed;

[0133] (2.4) Resource limits: Set strict limits on CPU time and memory usage to prevent infinite loop or memory exhaustion attacks.

[0134] If the feedback validator code executes successfully in the sandbox without triggering any permission violations and returns the expected reward value (such as 1.0 or 0.0) within the set resource limits (e.g., 100 milliseconds of CPU time, 50MB of memory), then it passes the verification of the dynamic execution steps. If the feedback validator code attempts to perform a violation (such as writing a file, accessing the network, etc.), it will be immediately terminated by the sandbox and reported as a failure. Similarly, if the code execution times out or exceeds the memory limit, it will also be judged as a failure, that is, it does not pass the verification of the dynamic execution steps.

[0135] In one specific implementation, the dynamic validator library described in this embodiment manages all feedback validators through versioning and indexing, facilitating rapid retrieval and invocation later. Specifically, this can be achieved by designing a table structure containing version information for the feedback validators in the dynamic validator library, making each feedback validator not just a record, but a version chain.

[0136] For example, a data model is established, which includes:

[0137] (1) A unique identifier (Verifier_id) represents a specific verification rule;

[0138] (2) Version number, which can be an auto-incrementing integer (1, 2, 3...);

[0139] (3) The specific executable code of this version;

[0140] (4) Provide feedback on the status of the validator, such as active, deprecated, etc.;

[0141] (5) Create timestamp (created_at);

[0142] The specific workflow is as follows: When a new feedback validator is created, the system assigns it a new unique identifier and marks it with a version number, such as version=1. When a user provides feedback to correct an existing rule (e.g., "No, the founder of Company A is now Wang Mou."), the server finds the unique identifier associated with that rule, marks the highest active version under that unique identifier as obsolete, inserts a new record using the same unique identifier but with the version number incremented by 1, stores the newly generated code, and sets its status to active. This design ensures the system's traceability, allows for the history of rule changes, and even enables rollback to older versions of rules when necessary.

[0143] In another specific implementation, to achieve rapid retrieval and invocation of feedback validators, this embodiment also includes an indexing step. Specifically, the original query or feedback text that triggers the generation of the feedback validator is converted into a high-dimensional vector using a sentence embedding model (such as text-embedding-v3), and this vector, along with a unique identifier, is stored in a dedicated vector database. When a new training prompt is sampled during subsequent optimization, it is similarly converted into a query vector. Then, an Approximate Nearest Neighbor Search is performed in the vector database to efficiently find the N feedback validators that are semantically most similar and relevant. In this way, when a simulated prompt is subsequently received, this embodiment can quickly filter out several relevant candidate validators from a large number of validators using the established index.

[0144] S206. When the background optimization process is triggered, obtain the simulated question instruction and obtain the candidate response generated by the intelligent agent model based on the simulated question instruction;

[0145] In this embodiment, the server can continuously and asynchronously run a background optimization process in the background. When the background optimization process is running, it samples a simulated question instruction (or prompt) from the server's problem log or a dedicated training dataset, and drives the intelligent agent model to generate one or more candidate responses based on the simulated question instruction.

[0146] When generating simulated question commands, the system can retrieve a massive log of all queries submitted by real users in the past, recorded by the server system, and sample seed questions from this log. Vector similarity retrieval can be used to obtain seed questions; specifically, the similarity threshold for vector similarity retrieval can be set relatively low (e.g., 0.6) to prevent questions from being too similar and to ensure question diversity.

[0147] S207. Based on semantic similarity, find and simulate candidate feedback validators related to the question instruction;

[0148] After obtaining candidate responses, all candidate feedback validators related to the current simulated question instruction are retrieved and executed based on the candidate responses. Specifically, the retrieval can be based on a pre-established index, and the specific method can be found in step S205 of this embodiment, which will not be repeated here.

[0149] S208. Verify and calculate the reward for the simulated questioning instruction based on the candidate feedback validator, and obtain the reward output value of each candidate feedback validator.

[0150] S209. The reward output values ​​of each candidate feedback validator are weighted and summed to obtain the reward signal. The policy parameters of the intelligent agent model are updated based on the reward signal to optimize the intelligent agent model.

[0151] In steps S208 and S209, after identifying candidate feedback validators, the simulated question instructions are validated and reward calculated to obtain the reward output value (e.g., 1.0 or 0.0) for each candidate feedback validator. Then, based on preset weights, the reward output values ​​of each candidate feedback validator are weighted and summed to obtain a reward signal. This reward signal is used to update the policy parameters of the intelligent agent model, thereby optimizing the intelligent agent model.

[0152] In a preferred embodiment, to improve training efficiency and stability, this embodiment preferably adopts the GRPO (Generalized Relative Policy Optimization) algorithm when optimizing training. It can calculate the advantage by normalizing the reward within a set of responses generated in a single session, thereby avoiding the drawback of the traditional PPO (Proximal Policy Optimization) algorithm, which requires training an additional independent value model with huge resource consumption, making it possible to perform continuous background training with limited resources.

[0153] Based on the above, it can be seen that this invention proposes an innovative closed-loop learning framework. Its core idea is to dynamically and automatically compile the user's natural language corrective feedback into a programmatic and verifiable reward function, thereby driving a continuous reinforcement learning with verifiable rewards (RLVR) process. This method differs from other existing technologies that passively collect user feedback as a one-time knowledge base patch, and it does not simply use feedback to generate corrective samples for imitation learning. Instead, it elevates each user correction into a reusable and generalizable reward function, thereby constructing a feedback verifier. Through the construction of multiple feedback verifiers, the model is taught a general ability to avoid similar errors, rather than simply memorizing an isolated correct answer.

[0154] The following example uses a restaurant's intelligent customer service model as an example to illustrate this concept. The restaurant recently launched a Super Member points redemption program with many rules, and only Super Members can redeem points for new products. However, according to information in the knowledge base, ordinary members can redeem points for any product currently on sale. When the first customer asks the intelligent customer service, "Can I redeem product C with my membership points?" (where the product is a new item), the ordinary intelligent agent model, based on information in the knowledge base, gives the feedback: "Member points can be redeemed for any product currently on sale." It is clear that this intelligent agent model gave an incorrect answer. In previous solutions, if the first customer gave the incorrect feedback: "Ordinary members cannot redeem product C with their membership points," the ordinary intelligent agent model would only remember this information. When the second customer asks, "Can I redeem product D with my membership points?" and product D is also a new item, it might still give an incorrect answer.

[0155] However, the intelligent agent model described in this embodiment, which can automatically optimize based on error feedback, upon receiving the feedback "Ordinary members cannot use membership points to redeem product C," will construct a feedback validator based on this feedback, generate automatically executed machine rules, and perform optimization training based on these rules. For example, the rules corresponding to the generated feedback validator could be: when a user asks a question involving product and points redemption, first check the user's membership type. If the intelligent agent model's answer contains the key information "Super Member Exclusive," it is judged as a "perfect answer" and given the highest reward; if the intelligent agent model's answer is ambiguous or outputs an incorrect result again, it is penalized or no reward is given. At this time, when there are no customer questions, the system can trigger the background optimization process, start the reinforcement learning optimizer, and automatically generate multiple related simulated question instructions, such as "Can I use points to redeem that new product?", "How do I redeem product E?", and "Is my membership level high enough to redeem product D?", etc. Based on these simulated question instructions, the intelligent agent model needs to answer the simulated questions in different ways, but each answer will be evaluated using the rules in the generated feedback validator, and the parameters of the intelligent agent model will be modified accordingly. After extensive practice and evaluation, the underlying neural network parameters of the intelligent agent model were continuously fine-tuned, so that the intelligent agent model no longer "searches" for the correct answer, but truly "learns" a new way of thinking: when encountering such problems, it must first check the membership level.

[0156] The method provided in this embodiment of the invention automatically generates generalizable verification rules by constructing a feedback validator, enabling the model to have a general optimization capability that automatically identifies and avoids a whole class of errors. This achieves a leap from existing single-point knowledge correction to policy-level optimization. Furthermore, by utilizing correction information from the user level, it solves the core bottleneck of existing reinforcement learning's over-reliance on experts, reducing dependence on expert information and achieving large-scale, low-cost generation of objective reward signals. Using the constructed feedback validator improves the objectivity of optimization and reduces model bias caused by user preferences. Moreover, the method provided in this embodiment can continuously and automatically learn and evolve while providing services, adaptively learning new knowledge and recording discovered error patterns, avoiding periodic, large-scale offline fine-tuning. Simultaneously, it maximizes the value of each user-provided error feedback, transforming a single feedback into a persistent digital asset that can be used infinitely for model training, significantly improving the efficiency of data and human capital utilization. Therefore, this method can automatically optimize the intelligent agent model based on user-provided error feedback, achieving the technical effect of adaptively improving the output accuracy of the intelligent agent model and enhancing its automation level.

[0157] The above describes the adaptive optimization method for the intelligent agent model based on error feedback information in the embodiments of the present invention. The following describes the adaptive optimization system for the intelligent agent model based on error feedback information in the embodiments of the present invention. Please refer to [link / reference]. Figure 3 One embodiment of the intelligent agent model adaptive optimization system based on error feedback information in this invention includes:

[0158] The feedback receiving module 301 is used to receive error feedback information from the user regarding the output content of the intelligent agent model from the feedback interface.

[0159] The verifier generation module 302 is used to obtain the verifier template corresponding to the error feedback information, generate a feedback verifier based on the content of the error feedback information and the verifier template, and store it in the dynamic verifier library.

[0160] The optimization execution module 303 is used to obtain a simulated question instruction when the background optimization process is triggered, and obtain the candidate response generated by the intelligent agent model based on the simulated question instruction. It searches for candidate feedback validators related to the simulated question instruction in the dynamic validator library, calls each candidate feedback validator to calculate the reward signal based on the candidate response, updates the strategy parameters of the intelligent agent model based on the reward signal, and optimizes the intelligent agent model.

[0161] The system provided in this embodiment of the invention can automatically optimize the intelligent agent model based on the error feedback information provided by the user, thereby achieving the technical effect of adaptively improving the output accuracy and automation level of the intelligent agent model.

[0162] In another embodiment of this application, the verifier generation module 302 is specifically used for:

[0163] The error feedback information is semantically parsed and intent identified to determine the correction intent type of the current error feedback information;

[0164] Based on the aforementioned correction intent type, the corresponding validator template is searched in the preset template library.

[0165] In another embodiment of this application, the types of correction intent include fact correction, calculation error correction, and formatting error correction.

[0166] In another embodiment of this application, the optimization execution module 303 is further configured to:

[0167] Based on semantic similarity, candidate feedback validators related to the question instruction are found and simulated;

[0168] Based on the candidate feedback validator, the simulated question instruction is verified and rewarded to obtain the reward output value of each candidate feedback validator;

[0169] The reward signal is obtained by weighted summation of the reward output values ​​of each candidate feedback validator.

[0170] In another embodiment of this application, the verifier generation module 302 is further configured to:

[0171] Perform feasibility testing on the generated feedback validator;

[0172] If the feedback validator passes the feasibility test, then the feedback validator is retained;

[0173] If the feedback validator fails the feasibility test, then the feedback validator is deleted.

[0174] In another embodiment of this application, the feedback receiving module 301 is further configured to:

[0175] The intelligent agent model is invoked to perform semantic analysis on the error feedback information to determine whether there is any ambiguity.

[0176] If there is any ambiguity, a clarifying question will be sent to the user based on the error feedback information, and the user's clarifying answer will be accepted.

[0177] Based on the clarification response, the ambiguous content of the error feedback information was corrected.

[0178] The system provided in this embodiment of the invention can run an asynchronous reinforcement learning loop in the background. This loop uses functions from a dynamic validator library to evaluate new responses generated by the model, thereby producing objective and accurate reward signals, and updating and optimizing the model based on these reward signals. Based on this, the intelligent agent model can be automatically optimized according to error feedback information provided by the user, achieving the technical effect of adaptively improving the output accuracy and automation level of the intelligent agent model.

[0179] Based on the same inventive concept, this specification also provides an electronic device with an intelligent agent model that adaptively optimizes based on error feedback information. The electronic device with an intelligent agent model that adaptively optimizes based on error feedback information in this embodiment of the invention will be described in detail below from the perspective of hardware processing.

[0180] Figure 4 This is a schematic diagram of an electronic device provided as an embodiment of this specification. Refer to the following... Figure 4 To describe the electronic device 400 according to this embodiment of the invention. Figure 4 The electronic device 400 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0181] like Figure 4 As shown, the electronic device 400 is presented in the form of a general-purpose computing device. The components of the electronic device 400 may include, but are not limited to: at least one processing unit 410, at least one storage unit 420, a bus 430 connecting different system components (including storage unit 420 and processing unit 410), a display unit 440, etc.

[0182] The storage unit stores program code that can be executed by the processing unit 410, causing the processing unit 410 to perform the steps described in the processing method section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 410 can perform, for example... Figure 1 or Figure 2 The steps are shown.

[0183] The storage unit 420 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 4201 and / or a cache storage unit 4202, and may further include a read-only memory unit (ROM) 4203.

[0184] The storage unit 420 may also include a program / utility 4204 having a set (at least one) program module 4205, such program module 4205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0185] Bus 430 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0186] Electronic device 400 can also communicate with one or more external devices 100 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 400, and / or with any device that enables electronic device 400 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 450. Furthermore, electronic device 400 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 460. Network adapter 460 can communicate with other modules of electronic device 400 via bus 430. It should be understood that, although... Figure 4 As not shown, other hardware and / or software modules may be used in conjunction with electronic device 400, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0187] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described in this invention can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this invention can be embodied in the form of a software product, which can be stored in a computer-readable storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, or network device, etc.) to execute the method described above according to this invention. When the computer program is executed by a data processing device, it enables the computer-readable medium to implement the method described above, i.e.: as... Figure 1 or Figure 2 The method shown.

[0188] Figure 5 This is a schematic diagram of a computer-readable medium provided for embodiments of this specification.

[0189] accomplish Figure 1 or Figure 2 The computer program of the method shown can be stored on one or more computer-readable media. A computer-readable medium can be a readable signal medium or a readable storage medium. A readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0190] The computer-readable storage medium may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0191] Furthermore, the present invention also provides a computer program product, including a computer program / instruction that, when executed by a processor, implements the intelligent agent model adaptive optimization method based on error feedback information as described in any of the above embodiments.

[0192] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0193] In summary, the present invention can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that in practice, general-purpose data processing devices such as microprocessors or digital signal processors (DSPs) can be used to implement some or all of the functions of some or all of the components according to the embodiments of the present invention. The present invention can also be implemented as a device or apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such programs implementing the present invention can be stored on a computer-readable medium or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0194] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the present invention is not inherently related to any specific computer, virtual device, or electronic device, and various general-purpose devices can also implement the present invention. The above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0195] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0196] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0197] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. An intelligent agent model adaptive optimization method based on error feedback information, characterized in that, The method comprises the following steps: receiving error feedback information of user on output content of intelligent agent model from feedback interface; performing semantic analysis and intention recognition on the error feedback information to determine the correction intention type of the current error feedback information, and based on the correction intention type, searching the corresponding verifier template of the error feedback information in a preset template library, wherein the verifier template is a parameterizable function skeleton; injecting parameters into the verifier template based on the information extracted from the content of the error feedback information to generate a feedback verifier and store it in a dynamic verifier library; when the background optimization process is triggered, obtaining a simulated questioning instruction and a candidate response generated by the intelligent agent model based on the simulated questioning instruction, searching for candidate feedback verifiers related to the simulated questioning instruction in the dynamic verifier library, and calling each candidate feedback verifier to calculate a reward signal based on the candidate response, updating the policy parameters of the intelligent agent model based on the reward signal, and optimizing the intelligent agent model. 2.The method of claim 1, wherein, The correction intention type includes factual correction, calculation error correction and format error correction. 3.The method of claim 1, wherein, The method comprises the following steps: based on semantic similarity, searching for candidate feedback verifiers related to the simulated questioning instruction; based on the candidate feedback verifiers, verifying the simulated questioning instruction and calculating the reward to obtain the reward output value of each candidate feedback verifier; weighting and summing the reward output values of each candidate feedback verifier to obtain the reward signal. 4.The method of claim 1, wherein, After generating the feedback verifier, the method further comprises the following steps: performing feasibility test on the generated feedback verifier; if the feedback verifier passes the feasibility test, retaining the feedback verifier; if the feedback verifier fails the feasibility test, deleting the feedback verifier.

5. The method of claim 1, wherein the error feedback information is obtained by using a method comprising: After receiving the error feedback information of user on output content of intelligent agent model from feedback interface, the method further comprises the following steps: calling the intelligent agent model to perform semantic analysis on the error feedback information to determine whether there is ambiguity; if there is ambiguity, sending a clarifying inquiry to the user based on the error feedback information and accepting the clarifying answer sent by the user; based on the clarifying answer, correcting the ambiguous content of the error feedback information.

6. An intelligent agent model adaptive optimization system based on error feedback information, characterized in that, The intelligent agent model adaptive optimization system based on error feedback information comprises: a feedback receiving module configured to receive error feedback information of user on output content of intelligent agent model from feedback interface; a verifier generating module configured to perform semantic analysis and intention recognition on the error feedback information to determine the correction intention type of the current error feedback information, and based on the correction intention type, search the corresponding verifier template of the error feedback information in a preset template library, wherein the verifier template is a parameterizable function skeleton; and inject parameters into the verifier template based on the information extracted from the content of the error feedback information to generate a feedback verifier and store it in a dynamic verifier library; The optimization execution module is configured to, when a background optimization process is triggered, acquire a simulated question instruction, acquire a candidate response generated by the intelligent agent model based on the simulated question instruction, find candidate feedback validators related to the simulated question instruction in the dynamic verifier library, and call each candidate feedback validator to calculate a reward signal based on the candidate response, update the policy parameters of the intelligent agent model based on the reward signal, and optimize the intelligent agent model.

7. An intelligent agent model adaptive optimization device based on error feedback information, characterized by, The error feedback information-based intelligent agent model adaptive optimization device comprises a memory and at least one processor, and the memory stores instructions; The at least one processor invokes the instructions in the memory to enable the error feedback information-based intelligent agent model adaptive optimization device to perform the steps of the error feedback information-based intelligent agent model adaptive optimization method according to any one of claims 1-5.

8. A computer-readable storage medium having stored thereon computer programs / instructions, characterized in that, The program / instructions enable the processor to perform the steps of the error feedback information-based intelligent agent model adaptive optimization method according to any one of claims 1-5 when the program / instructions are executed by the processor.

9. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instructions enable the processor to perform the steps of the error feedback information-based intelligent agent model adaptive optimization method according to any one of claims 1-5 when the computer program / instructions are executed by the processor.

Images