Intelligent agent training method, data processing method and related device

Abstract

The invention discloses an agent training method, a data processing method and a related device, and relates to the field of artificial intelligence. In an agent training process, key information in target interaction data generated by interaction between an initial agent and a target user is automatically extracted to generate a structured memory; and a context basis for supporting training can be constructed without manually annotating data, so that the training cost of the intelligent agent is reduced. Traceable historical context support is provided for the intelligent agent by processing the target interaction data and updating the target database based on the processing result; a comprehensive basis is provided for agent behavior evaluation by acquiring whole-process trajectory data generated when an initial agent executes a task in a target task environment; the scoring model constructed by the user for the feedback information of the full interaction trajectory scores the trajectory data, and the initial agent parameters are updated, so that the agent can continuously optimize the strategy, and the accuracy of the agent is effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to an agent training method, a data processing method and related devices. BACKGROUND

[0002] With the development of large language models (LLM, referred to as large models), AI agents (AI Agent) based on LLM are widely used in the processing of various tasks. At present, the optimization and training of agents mainly have the following problems: model optimization is heavily dependent on a large amount of high-quality artificial annotation data, which is costly and difficult to adapt to changes; a complex reward function needs to be designed in advance, which relies on expert knowledge and is prone to training bias; and there is a lack of dynamic adjustment of individualized ability. Therefore, how to efficiently train and optimize such agents to stably and reliably process complex tasks is still a problem to be solved. SUMMARY

[0003] In view of the above problems, the present application provides an agent training method, a data processing method and related devices to reduce the cost of agent training and improve the accuracy of the agent. The specific solutions are as follows:

[0004] The first aspect of the present application provides an agent training method, comprising:

[0005] Collecting target interaction data generated by an initial agent interacting with a target user;

[0006] Processing the target interaction data and updating a target database based on the processing result, the target database being used to store structured historical context information;

[0007] Obtaining trajectory data generated by the initial agent executing a corresponding target task in a target task environment, the trajectory data representing complete process record data of the initial agent completing a complete target task;

[0008] Scoring the trajectory data based on a scoring model to obtain a scoring result, the evaluation of the scoring model being realized based on user feedback signals extracted from the trajectory data;

[0009] Optimizing policy data of the initial agent based on the scoring result to obtain a target agent, the target agent being configured to execute a task based on the optimized policy data.

[0010] In a possible implementation, the processing of the target interaction data and the updating of the target database based on the processing result comprises:

[0011] In a case where the target interaction data meets a condition of being converted into long-term memory, target data in the target interaction data is extracted, the target data at least including target user information, a target user intention, a processing flow executed by the initial agent in an interaction process, and a processing result generated by the initial agent based on the processing flow;

[0012] The target data is encapsulated in a structured format of data in a target database, to generate structured memory content;

[0013] The target database is updated based on the structured memory content.

[0014] In a possible implementation, the target database includes a key-value storage subsystem and a vector retrieval subsystem, and updating the target database based on the structured memory content includes:

[0015] The structured memory content is stored in the key-value storage subsystem;

[0016] After a core text field in the structured memory content is converted into a vector, the structured memory content is stored in the vector retrieval subsystem;

[0017] Retrieving target historical context information corresponding to the target task environment in the target database includes:

[0018] A vector representation of a core semantic feature of the target task environment is calculated;

[0019] In the vector retrieval subsystem, a target vector corresponding to the vector representation of the core semantic feature and target structured memory content corresponding to the target vector are retrieved;

[0020] According to identification information recorded in the target structured memory content, an interaction history record is retrieved from the key-value storage subsystem;

[0021] The interaction history record and the target structured memory content are fused to obtain target historical context information.

[0022] In a possible implementation, the scoring result is obtained by scoring the trajectory data based on the scoring model, and the scoring includes:

[0023] Historical context information related to the target task environment is retrieved from the target database;

[0024] The historical context information, a completion condition corresponding to the target task, and the trajectory data are input into the scoring model to obtain a scoring result.

[0025] In a possible implementation, the optimizing the policy data of the initial agent based on the score result to obtain a target agent comprises:

[0026] According to the score result, a relative merit ranking of each piece of trajectory data is determined;

[0027] According to the relative merit ranking, a reward signal corresponding to each piece of trajectory data is generated; the reward signal is used to guide the optimization direction of the policy data of the model;

[0028] The policy data of the initial agent is optimized according to the reward signal to obtain a target agent, so that the target agent outputs trajectory data satisfying a high reward signal.

[0029] The second aspect of the present application provides a data processing method, applying a target agent, comprising:

[0030] Receiving query information of a user;

[0031] According to the query information, retrieving associated target historical interaction data in a target database, wherein the target database stores structured memory content updated iteratively based on interaction data in the training process of the target agent;

[0032] Based on the target historical interaction data, dynamically constructing prompt information for processing the current query;

[0033] Inputting the query information and the prompt information into the target agent to obtain target output information corresponding to the query information.

[0034] In a possible implementation, the dynamically constructing prompt information for processing the current query based on the target historical interaction data comprises:

[0035] In a case where at least one question and answer pair is parsed from the target historical interaction data, inserting the question and answer pair as a sample example into a prompt word template to obtain prompt information for processing the current query.

[0036] In a possible implementation, the inserting at least one question and answer pair in the target historical interaction data as a sample example into a basic prompt word to obtain prompt information for processing the current query comprises:

[0037] Calculating a semantic vector representation of the query information;

[0038] Based on the semantic vector representation of the query information, performing retrieval in the target database to obtain the target historical interaction data matching the semantics of the query information;

[0039] In a case where the target historical interaction data is parsed to obtain at least one question-answer pair, the question-answer pair is inserted into a preset prompt word template to obtain prompt information for processing the current query.

[0040] The third aspect of the application provides an agent training apparatus, comprising:

[0041] The acquisition unit is configured to acquire target interaction data generated by interaction between an initial agent and a target user;

[0042] The first updating unit is configured to process the target interaction data and update a target database based on a processing result, the target database being configured to store structured historical context information;

[0043] The first acquisition unit is configured to acquire trajectory data generated by the initial agent when performing a corresponding target task in a target task environment, the trajectory data representing complete-process record data of the initial agent when completing a target task;

[0044] The scoring unit is configured to score the trajectory data based on a scoring model to obtain a scoring result, the scoring model being evaluated based on feedback of a user on the trajectory data;

[0045] The second updating unit is configured to optimize strategy data of the initial agent based on the scoring result to obtain a target agent, the target agent being configured to perform a task based on the optimized strategy data.

[0046] The fourth aspect of the application provides a data processing apparatus, comprising:

[0047] The receiving unit is configured to receive query information of a user;

[0048] The retrieving unit is configured to retrieve associated target historical interaction data from a target database based on the query information, the target database storing structured memory content updated iteratively based on interaction data in a target agent training process;

[0049] The generating unit is configured to dynamically construct prompt information for processing the current query based on the target historical interaction data;

[0050] The second acquisition unit is configured to input the query information and the prompt information into the target agent to obtain target output information corresponding to the query information.

[0051] By means of the technical solutions, the application provides an agent training method, a data processing method and related devices. In the training process of the agent, key information in target interaction data generated by interaction between an initial agent and a target user is automatically extracted to generate structured memory, a context basis supporting training is constructed without relying on a large amount of manually annotated data, and the training cost of the agent is reduced. The structured storage and continuous accumulation of the interaction data are realized by processing the target interaction data and updating the target database based on the processing result, the agent is provided with traceable historical context support; the overall basis for behavior evaluation of the agent is provided by acquiring whole-process trajectory data generated by the initial agent in the target task environment when performing a task; the trajectory data is scored by a scoring model constructed based on feedback information of the user on the whole interaction trajectory, and automatic and objective evaluation of the behavior of the agent is realized; and finally, the initial agent parameters are updated based on a reward signal corresponding to the scoring result, so that the agent can continuously optimize its strategy, and finally obtain a target agent capable of outputting high-quality behavior trajectory, and the accuracy of the agent is effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0052] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals are used to represent the same or similar elements. It should be understood that the drawings are schematic, and the original and elements are not necessarily drawn according to the scale.

[0053] Figure 1 A flowchart of an agent training method provided by the application;

[0054] Figure 2 A flowchart of a data processing method provided by the application;

[0055] Figure 3 A flowchart of an agent training and use provided by the application;

[0056] Figure 4 A flowchart of an agent reinforcement learning training provided by the application;

[0057] Figure 5 A flowchart of construction of a large model context window based on multi-module input in an application scenario provided by the application;

[0058] Figure 6 A structural diagram of an agent training device provided by the application;

[0059] Figure 7 A structural diagram of a data processing device provided by the application. DETAILED DESCRIPTION

[0060] The embodiments of the present application are described below in conjunction with the accompanying drawings. The terms used in the implementation part of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0061] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art can know that with the development of technology and the emergence of new scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0062] The terms "first", "second", and the like used in the present application are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged under appropriate circumstances, which is only a distinguishing way adopted in describing the embodiments of the present application for the objects with the same attribute in description. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, so that the process, method, system, product or equipment containing a series of units does not have to be limited to those units, but can include other units not clearly listed or inherent to these processes, methods, products or equipment.

[0063] The embodiments of the present application provide an agent training method, which collects interaction data of a user and an initial agent, processes and updates a target database for storing structured historical context; obtains trajectory data of the initial agent performing a target task, scores the trajectory data by using a scoring model constructed based on context information retrieved from the target database; updates parameters of the initial agent based on a reward signal of the scoring result to obtain a target agent. A data processing method generates prompt information based on the target agent trained and the historical context retrieved from the target database, so as to realize accurate response to user queries. The scheme can be applied to scenarios such as intelligent customer service (e.g., optimizing problem solving based on user historical consultation records), business assistants of enterprises (e.g., optimizing task execution process in combination with past business operation records of employees), etc., significantly reducing dependence on manual annotation, and improving generalization ability and iteration efficiency of the agent.

[0064] Referring to Figure 1 A flowchart of an agent training method provided by the embodiments of the present application is shown. The method can include the following steps:

[0065] S101, collect target interaction data generated by interaction between an initial agent and a target user.

[0066] The target user refers to the user currently interacting with the initial agent, which is a pre-trained generated agent and can be an agent used in the current application scenario. In order to make the output of the agent more accurate, the initial agent can be iteratively trained and updated in real time to obtain a more accurate target agent. The target interaction data represents the complete record data generated by the initial agent in the actual application scenario in multiple rounds of dialogue with the target user, such as including user queries, agent responses, tool call results, user feedback information (such as information generated by user likes, corrections, and selections), etc.

[0067] The input and output of the agent API interface can be recorded in real time through the log system. For example, in a customer service scenario, the user asks "how to reset the password", the agent calls the knowledge base query tool and returns the answer steps, and the user's final satisfaction score is recorded. These record data are used as target interaction data. These target interaction data provide a real, massive and diversified data basis for subsequent construction of a memory library (i.e., a target database), avoiding the high-cost and inefficient manual data labeling process.

[0068] Correspondingly, in the process of training the initial agent, the required computing resources (backend) and model basis (such as the model structure of the initial agent) for training can be selected and initialized. For example, in the training platform configuration, specify which GPU machines to use, select a base model (such as Qwen-7B), and set hyperparameters (such as learning rate). The platform initializes a training task and loads the base model into the GPU.

[0069] S102, processing the target interaction data and updating the target database based on the processing result.

[0070] The target database is used to store structured historical context information, i.e., the target database is a shared database for storing structured historical context information, which serves as the core carrier of the agent's memory. Among them, structured means that data is organized according to a pre-set Schema (such as JSON format), including user ID, timestamp, content type, memory content, etc. A large language model (LLM) can be used as a filter to analyze each round of interaction. For example, the LLM determines that the user's preference information "I prefer to reply in English" has long-term value, and then stores it in the database after being structured. Unstructured interaction data is converted into structured knowledge that can be efficiently queried and utilized by machines, providing context support for agent training and reasoning.

[0071] S103, obtaining trajectory data generated by the initial agent in the target task environment when performing the corresponding target task.

[0072] The trajectory data represents the complete record of the initial agent completing a complete target task, which includes a complete record of a series of decisions and actions taken by the agent in the target task environment to complete the target task. The target task environment represents the scenario conditions when the initial agent performs the target task, including the task type (such as order query, data statistics), task constraint conditions (such as response time limit, data source range), input and output format requirements, etc.

[0073] In the training framework, the initial agent is allowed to try to complete the same task (such as "generating a financial report analysis summary for a user") multiple times in a virtual environment, and the thinking (Thought), action (Action), observed result (Observation), and final task completion degree (Reward) at each step are recorded completely. The trajectory data can comprehensively reflect the pros and cons of the strategy of the initial agent in processing the target task, and provide a data basis for subsequent automated evaluation.

[0074] S104, scoring the trajectory data based on the scoring model to obtain a scoring result.

[0075] The evaluation of the scoring model is based on the user feedback signal extracted from the trajectory data, wherein the user feedback signal includes direct feedback signal of the user interacting with the initial agent and / or indirect feedback signal generated by inferring the user satisfaction based on the context information and processing result of the target interaction data. The scoring model can be an independent large language model, the input of which includes the trajectory data of the user interacting with the initial agent, and the scoring result is output by analyzing the direct feedback (such as likes and dislikes) or indirect feedback (such as user satisfaction inferred from the context of the dialogue) of the user in the trajectory data. The scoring model of the embodiment of the present application can generate high-quality indirect feedback signals autonomously by analyzing the logical consistency of the interaction context and the processing result, the task completion degree, and other objective indicators without relying on direct user feedback. This capability greatly reduces the dependence on manual annotation, enabling the agent training process to be automated and continuously optimized in a user-unaware or non-active feedback scenario. Specifically, the scoring model can be an analyzer constructed based on a large language model, and its core function is to evaluate the relative pros and cons of the behavior trajectory of the agent. For example, the scoring model is prompted to "sort and score these solutions according to the quality and efficiency of task completion" by inputting multiple pieces of trajectory data and task requirements into the scoring model, and retrieving successful cases (such as intelligent dialogues with high user ratings) corresponding to the target task from the target database as scoring references. In the embodiment of the present application, the scoring model replaces the cumbersome process of manually designing the reward function, and the evaluation standard is more flexible and closer to human preferences, and can make more accurate judgments combined with historical experience.

[0076] S105, based on the scoring result, optimizing the policy data of the initial agent to obtain a target agent.

[0077] The scoring result is a result obtained by scoring the trajectory data by the scoring model. The reward signal can be converted from the scoring result, and the reward signal is used to update the policy data of the initial agent. The higher the score, the stronger the reward signal, and vice versa. The policy data can include the policy analyzed by the user input and the related processing parameters in the model, such as the parameter representing the core parameter affecting the output result in the initial agent, such as the weight and bias of the neural network. The parameter update can adjust the decision logic and output policy of the agent. According to the ranking given by the scoring model, high reward values are assigned to trajectories with high rankings, and low reward values are assigned to trajectories with low rankings. Then, using a reinforcement learning algorithm (such as PPO), the model parameters are adjusted according to these reward signals to make them more likely to produce high-reward behaviors in the future. The automation and targeted optimization of the agent's policy are achieved, and a more powerful and reliable target agent is ultimately obtained.

[0078] For example, taking the training of an enterprise financial agent as an example, the target user is an enterprise financial personnel, and the target task is to "generate a quarterly tax declaration detail table". First, collect the interaction data between the financial personnel and the initial agent (such as the financial personnel inputting "need to include value-added tax and corporate income tax data", the agent outputting "value-added tax data has been obtained, corporate income tax data is missing", and the financial personnel feeding back "query corporate income tax data from the ERP system"), and update the target database after processing; then obtain the trajectory data of the initial agent executing the "generate a quarterly tax declaration detail table" task (including calling the ERP system to query data, data integration, generating a detail table, format adjustment, etc.); retrieve the historical context information of the past "generate a tax report" from the target database (such as the format requirements of the past declaration detail table, the data source priority), build a scoring model to score the trajectory data (such as deducting 10 points for not calling the ERP system to query data according to the priority, and finally getting 80 points); update the initial agent parameters based on the reward signal corresponding to 80 points (such as adjusting the "data source calling priority" related parameters), and obtain the target agent.

[0079] After the training of the agent is completed, the target agent can be finally verified and applied to the corresponding application platform. For example, running the trained model on an independent test set, calculating key indicators (such as task success rate, average reward). After meeting the standard, the model file (such as LoRA weight) is packaged and released as a callable API service through the platform's deployment system. This forms a complete closed loop from training to application, making the target agent better applied.

[0080] The embodiment of the application provides an agent training method, in the training process of the agent, key information in target interaction data generated by interaction between an initial agent and a target user is automatically extracted to generate structured memory, a context basis supporting training can be constructed without relying on a large amount of manually annotated data, and the training cost of the agent is reduced. Through processing of the target interaction data and updating of a target database based on a processing result, structured storage and continuous accumulation of the interaction data are realized, and traceable historical context support is provided for the agent; through acquisition of whole-process trajectory data generated by the initial agent in a target task environment when performing a task, comprehensive basis is provided for behavior evaluation of the agent; a scoring model constructed by feedback information of the user on the whole interaction trajectory is used to score the trajectory data, and automatic and objective evaluation of the behavior of the agent is realized; finally, an initial agent parameter is updated based on a reward signal corresponding to a scoring result, so that the agent can continuously optimize the strategy, and finally a target agent capable of outputting a high-quality behavior trajectory is obtained, and the accuracy of the agent is effectively improved.

[0081] The training process of the agent in the embodiment of the application will be described below in combination with a specific application scenario.

[0082] In some embodiments of the application, the processing process of processing the target interaction data and updating the target database based on the processing result comprises: in the case that the target interaction data meets the condition of being converted into long-term memory, target data in the target interaction data is extracted; the target data is encapsulated according to a structured format of data in the target database to generate structured memory content; and the target database is updated based on the structured content.

[0083] The target data at least comprises target user information, a target user intention, a processing flow executed by the initial agent in the interaction process and a processing result generated by the initial agent based on the processing flow, wherein the target user information represents information that can identify the identity or demand preference of the target user, such as user occupation, historical demand type and the like; the target user intention represents the core purpose of the user interacting with the agent, such as querying information, requesting operation, obtaining suggestion and the like; the processing flow executed by the initial agent in the interaction process and the processing result generated by the initial agent based on the processing flow represent the strategy and processing process of the agent in the historical interaction process of the user. The condition of being converted into long-term memory represents a standard for judging whether the target interaction data has long-term storage value, such as whether the data contains complete user intention, whether it is a new demand first appearing, whether the user feedback is clear and the like.

[0084] After the target data is filtered from the target interaction data, the target data needs to be packaged in a structured format to obtain structured memory content, wherein the structured format is a predefined data schema, which specifies the organization of the memory content, and usually includes metadata (such as user identification user_id, timestamp) and content data (such as query, response, feedback). Unifying data into a standardized data structure facilitates subsequent efficient retrieval, management and application, and is the basis for building a target database. The generated JSON object is written into the database. The subsequent embodiments will detail the specific storage method. In the embodiments of the present application, the interaction data is filtered according to the condition of being converted into long-term memory, ensuring that the content in the target database is high-value information that is filtered and structured, providing a reliable data foundation for the evolution of the intelligent agent. Specifically, for information that is valuable in a single round of dialogue but does not need to be permanently stored (such as a temporary calculation result), it can be attached to the current session state object as a "note" and only valid within the session period. It can also be analyzed using a lightweight model or rule engine. The basis for judgment can include: whether it contains explicit user feedback (such as likes, adoption); whether it successfully completes a complex task; whether it contains a user's explicit preference statement (such as "I prefer to use English"); whether it corrects the previous error of the intelligent agent, etc.

[0085] Correspondingly, after the structured memory content is generated, it can be made to be persistently stored in the target database, for example, for long-term memory content, records are generated according to a fixed schema. For example, a piece of memory can include the fields of session_id, user_id, agent_id, type (such as user_preference, successful_qa), content (specific content), embedding (vector), etc.

[0086] In some embodiments of the present application, the target database includes a key-value storage subsystem and a vector retrieval subsystem, wherein based on the structured memory content, the updating process of the target database includes: storing the structured memory content in the key-value storage subsystem; after converting the core text field of the structured memory content into a vector, storing it in the vector retrieval subsystem.

[0087] Correspondingly, the process of retrieving the target historical context information corresponding to the target task environment in the target database comprises: calculating the vector representation of the core semantic features of the target task environment; in the vector retrieval subsystem, retrieving the target vector corresponding to the vector representation of the core semantic features and the corresponding target structured memory content; according to the identification information recorded in the target structured memory content, retrieving the interaction history record from the key-value storage subsystem; and fusing the interaction history record and the target structured memory content to obtain the target historical context information.

[0088] In this embodiment, the target database comprises a key-value (KV) storage and a vector retrieval subsystem. The key-value storage subsystem represents a database submodule adopting a key-value (Key-Value) storage structure, wherein the "key" is the index information (such as the combination of "user ID - interaction timestamp") uniquely identifying the structured memory content, and the "value" is the complete structured memory content (including the full fields of user features, intent, key context, agent response, etc.). The core function of this subsystem is to realize accurate storage and fast retrieval of structured memory content, ensuring the integrity of the context information. The vector retrieval subsystem represents a submodule constructed by a vector database. By converting the text field into a high-dimensional vector (Embedding), the vector similarity calculation realizes "fast matching at the semantic level", and the core function is to quickly filter out the context data related to the semantic of the target task environment from the massive data. The key-value storage structure has the characteristic of "fast positioning by Key", and can complete the retrieval of a single structured memory content in milliseconds, solving the problem of "slow multi-table association query" of traditional relational databases. The complete storage of the full fields of the structured memory content ensures that the key context information will not be missed during subsequent retrieval, providing complete basis for agent evaluation. The core text field in the structured memory content represents the field in the structured memory content that best reflects the interaction semantics (such as user intent description, key context details, task requirement keywords, etc.), which is the core basis for judging the semantic association of different interaction data. Converting the text semantics into vector storage solves the problem that traditional "keyword matching retrieval" cannot identify synonyms and semantically similar expressions (such as "booking a conference room" and "applying for a conference site" are semantically similar and have high vector similarity, which can be effectively matched); the vector retrieval subsystem supports the retrieval of millions of data in milliseconds, greatly improving the semantic matching efficiency in massive interaction data, and laying a foundation for the fast screening of target historical context information.

[0089] It should be noted that after obtaining the target database in the embodiments of the present application, the size of the memory bank can be effectively controlled while retaining the core knowledge by summarizing and pruning, thereby ensuring the retrieval efficiency and accuracy. Among them, the processing mode based on summarization compression refers to using a language model to summarize multiple similar or continuous memory contents, and replacing multiple original memories with one more refined memory. For example, a background task is periodically run to cluster memories of the same topic (such as multiple records about the time stamp timezone under the same user identification user_id), and then use an LLM to generate a summary memory (such as “the user confirms multiple times that his time zone is EST”), and delete or archive the original records. In this way, the number of memories can be significantly reduced, the storage overhead and retrieval noise can be reduced, and higher-order knowledge can be refined. The processing mode based on pruning refers to actively deleting memories with low value or outdated according to a preset strategy. The pruning strategy can be defined, such as an LRU strategy: deleting memories that have not been accessed for the longest time; a low-frequency strategy: deleting memories with an access frequency below a certain threshold; a time-sensitive strategy: setting a TTL (time to live) for memories, and automatically deleting expired memories. In this way, the memory bank can be prevented from expanding indefinitely, the performance of the system during long-term operation can be ensured, and outdated information can be automatically eliminated.

[0090] The core semantic features of the target task environment represent the semantic information (such as task name, task requirement description, execution condition, etc.) that best reflects the task type, requirement scenario, and constraint condition in the target task environment. Vector representation of the core semantic features can convert the core semantic features into high-dimensional vectors stored in the vector retrieval subsystem, ensuring the feasibility of similarity calculation. For example, a retrieval request is sent to the vector retrieval subsystem, and the request parameters include “reference vector”, “similarity threshold 0.7”, “return Top5 results”, and other information; the vector retrieval subsystem calculates the cosine similarity between the reference vector and all vectors in the library, and selects the top 5 target vectors (such as similarity 0.85, 0.82, 0.78, 0.75, 0.72); extract the Key associated with the 5 target vectors (such as “user_003-20240605162000” “user_005-20240608101500” and the like), and retrieve the corresponding structured memory content from the key-value storage subsystem as the target structured memory content.

[0091] The identification information recorded in the target structured memory content is information (such as a user ID, an interaction session ID, etc.) representing the complete interaction process of the user that can be associated with the target structured memory content, and the core role is to expand the "single context data" into "complete interaction link data of the user"; the "interaction history record" represents the complete interaction process data (such as the interaction record of a user applying for a conference room multiple times, the response record of an intelligent agent processing the same type of booking demand multiple times, etc.) of the user in the past related to the target task environment, and is a supplement and expansion of the target structured memory content. In the process of fusing the interaction history record and the target structured memory content, the information can be integrated according to a preset rule, such as removing redundant information and supplementing associated information, to form a logically coherent and complete context set, which not only retains the core information related to semantics, but also supplements the complete interaction link and user preferences, ensuring that the target historical context information can fully reflect the past processing experience of the same type of task; the scoring model based on the information has a more practical business scenario and user demand, avoiding one-sidedness in model evaluation due to incomplete information.

[0092] In the embodiments of the present application, the scoring model can be applied for evaluation, so that the evaluation result is more objective and has interpretability. Correspondingly, the trajectory data is scored based on the scoring model to obtain a scoring result, including: retrieving the historical context information related to the target task environment from the target database; inputting the historical context information, the completion condition corresponding to the target task, and the trajectory data into the scoring model to obtain the scoring result. The historical context information can include: stored successful or failed interaction cases (QA pairs); long-term preferences and intentions of the user, past task processing processes and results. These information provides experience basis or reference standard for the scoring model, rather than being processed by fixed scores. It can be seen that the scoring model in the embodiments of the present application is not a simple query output tool, but a model that can understand task requirements, refer to historical experience, analyze current trajectory and make independent judgments. Specifically, the historical context information refers to the scoring records of the intelligent agent trajectory in the past similar task environment, including trajectory data content, scoring result, scoring reason, etc. These data can be stored in the target database as part of the structured memory content. The vector of the current task environment can be calculated, and then the historical scoring records with similar semantics (for example, scoring of "financial report analysis" tasks) are searched in the vector library. In the embodiments of the present application, the input to the scoring model can include not only the trajectory data but also the completion condition corresponding to the target task and the historical scoring data, which makes the scoring process a verifiable comparison and analysis process. Correspondingly, the completion condition corresponding to the target task and the historical scoring data can be generated into a prompt word, and the prompt word is input into the scoring model, so that the scoring model scores the trajectory data based on the prompt word.

[0093] For example, in the process of training the intelligent agent for code review, a code modification suggestion (trajectory data) proposed by an intelligent agent needs to be scored. Several high-score and low-score modification suggestions and their scoring reasons for the task of "fixing code vulnerabilities" in history can be retrieved from the target database. At this time, the prompt word "task: fix the **vulnerability of the Frontend page. Historical high-score case: suggestion 1 (95 points): parameterized query is used to completely eliminate vulnerabilities. Historical low-score case: suggestion 2 (60 points): only character filtering is done, the solution is incomplete. To be evaluated suggestion: suggestion 3, please score." is input into the scoring model. The scoring model refers to the historical standards and finds that the to-be-evaluated suggestion is more similar to the high-score case, so it gives a score of 92 and states that "this suggestion is similar to high-score case 1, which adopts a root cause repair solution."

[0094] In the embodiments of the present application, scoring is performed by the scoring model, so that the behavior of the entire training system is more stable and predictable, significantly improving the effect and efficiency of reinforcement learning training.

[0095] In some embodiments of the present application, the policy data of the initial intelligent agent is optimized based on the scoring result to obtain a target intelligent agent, including: determining the relative advantage and disadvantage order of each piece of trajectory data according to the scoring result; generating a reward signal corresponding to each piece of trajectory data according to the relative advantage and disadvantage order; the reward signal is used to guide the optimization direction of the policy data of the model; and the policy data of the initial intelligent agent is optimized according to the reward signal to obtain a target intelligent agent, so that the target intelligent agent outputs trajectory data satisfying a high reward signal.

[0096] In this embodiment, the relative advantage and disadvantage order refers to not paying attention to the absolute numerical value of the score, but paying attention to the ranking order between multiple trajectories. For example, in three trajectories, it is determined which one is first, which one is second, and which one is third, so that the ranking is more intuitive and stable, and is not easily affected by individual bias of the scoring model. According to the relative advantage and disadvantage order, a reward signal corresponding to each piece of trajectory data is generated, wherein the reward signal is used to guide the optimization direction of the model parameters, which can be a scalar value, and the size is determined by the ranking of the trajectory. The higher the ranking, the greater the reward value. For example, the Elo rating system is used to update the "score" of each strategy according to the ranking, and the score change is used as the reward; or more simply, the first-ranked trajectory is assigned a reward of +1, the last-ranked trajectory is assigned a reward of -1, and the intermediate ones are assigned a reward of 0.

[0097] Then a reinforcement learning algorithm such as Proximal Policy Optimization (PPO) can be used. The algorithm calculates the probability of obtaining a high reward under the current policy, and then adjusts the model parameters through gradient ascent to maximize the future expected reward. In this way, the agent's policy is automatically and iteratively optimized. After several rounds of "generating trajectories-scoring-assigning rewards-updating parameters", the initial agent's policy is constantly improved, and the final performance is significantly improved, becoming a target agent that can stably output high-quality behaviors.

[0098] In order to enable the agent to meet different task requirements, the target task environment of the agent can also be constructed, and by retrieving relevant data sets from the database and dynamically inserting prompt words, a unique and context-rich background information is constructed for each training task, thereby greatly improving the diversity and quality of training data. Specifically, the data set corresponding to the training target can be retrieved from the target database according to the training target of the initial agent; the data set is inserted into the basic prompt word of the task environment to construct a context-containing target task environment.

[0099] The training target refers to which aspect of the agent's ability the current training hopes to strengthen, such as "improve the efficiency of handling customer complaint emails". The corresponding data set refers to structured memory content related to the target, such as historical successful cases, common problems, and user portraits. The basic prompt word defines the basic template of the task, for example, the Few-shot prompt word construction technology can be used. The retrieved data set (such as 3 successful complaint handling cases) is spliced after the basic prompt word. For example: "You are a customer service AI. Please handle the following customer email. Reference cases: <case 1>, <case 2>, <case 3> Now please handle the new email: <user email>". In this way, a context-containing target task environment is obtained, that is, a virtual task environment that is enhanced by historical cases and knowledge and is more specific and complex. By dynamically constructing a context-rich training environment, the problem of disconnection between the simulation environment and the real environment is solved. It enables the agent to be exposed to various real scenarios and solutions during training, greatly improving the training efficiency and the adaptability and robustness of the trained agent in the real world.

[0100] It should be noted that, in order to ensure that the data between different users or different agents does not interfere with each other, and to protect privacy and achieve personalized services, in the case of multiple agents or multiple users sharing a corresponding target database, the related context can be isolated. The data in the target database adds metadata identification, which is a field indicating the ownership of the structured memory content, such as user identification user_id or agent identification agent_id. Correspondingly, the metadata identification can be used for retrieval, that is, when retrieving information from the memory library, the user / agent identification is used as a mandatory filtering condition. For example, after vector retrieval or KV retrieval, a filtering condition is added: user_id = current_user_id AND agent_id = current_agent_id. In this way, only the memory of the current user and the current agent can be retrieved. Thus, strict memory isolation is achieved. User A cannot see the preferences of user B, and the customer service agent cannot see the internal knowledge of the technical support agent, thus ensuring privacy and security and providing a truly personalized experience.

[0101] In an embodiment of the present application, a method for applying a trained target agent to process data is also provided, see Figure 2 which shows a flowchart of a data processing method. The method can include the following steps:

[0102] S201, receiving query information of a user.

[0103] S202, retrieving associated target historical interaction data in the target database according to the query information.

[0104] The target database stores structured memory content updated iteratively based on interaction data during the training of the target agent.

[0105] S203, dynamically constructing prompt information for processing the query based on the target historical interaction data.

[0106] S204, inputting the query information and the prompt information into the target agent to obtain target output information corresponding to the query information.

[0107] The text, voice and other information input by the user can be determined as the query information of the user through the API interface of the target agent, the chat window or other interactive forms. Then the semantic vector of the user query is calculated, and all the historical interactions, user preferences, successful cases and other information most relevant to the current query are retrieved as the target historical interaction data through the collaborative work of the vector library and the KV library. Thus, the processing and output process of the agent for the query information of the user is based on all the historical experience, and the personalization (such as remembering the user preferences) and precision (referring to the successful cases) of the answer are realized, which significantly improves the user experience.

[0108] Then, based on the target historical interaction data, prompt information for the current query is dynamically constructed. In one embodiment of the present application, in the case that at least one question and answer pair is obtained by analyzing the target historical interaction data, the question and answer pair is inserted into the prompt word template as a sample example to obtain the prompt information for processing the current query. The prompt information is a prompt word containing task instructions, historical context and examples, which is used to guide the output of the target agent, for example, the historical context and information retrieved from the database are assembled into a new target prompt information according to a preset template. For example, the three most relevant QA pairs retrieved are inserted into the system prompt as Few-shot examples. Then the target prompt information and the query information of the user are input into the target agent together, so that the target agent generates the final target output information based on all the context information. The target agent of the present embodiment has a real "memory" and "context awareness" capability, and it provides not isolated, static answers, but continuous, personalized intelligent services, which greatly enhances the practicality and user experience effect of the product.

[0109] The target agent of the present embodiment has a real knowledge memory and context awareness capability, and it provides not isolated, static answers, but continuous, personalized intelligent services, which greatly enhances the practicality and user experience effect of the product. Accordingly, in the case that a suitable question and answer pair cannot be parsed from the target historical interaction data (for example, when facing a new user or a new type of problem), the system has a related processing mode. For example, at this time the system will use the preset basic prompt word, or combine the retrieved other non-exemplary context information (such as user portrait, general knowledge fragments, etc.) to construct the prompt information for the current query. This design ensures that the system can still maintain basic reliability and usability in the cold start stage or when processing unknown tasks. With the continuous interaction and the continuous enrichment of the memory library, the system will dynamically and smoothly transition from relying on basic capabilities to relying on enhanced historical context and examples, thereby realizing the continuous evolution of the quality of the agent service.

[0110] In some embodiments of the present application, the at least one question and answer pair in the target historical interaction data is inserted into the base prompt as a sample example to obtain prompt information for processing the current query, including: calculating a semantic vector representation of the query information; based on the semantic vector representation of the query information, searching in the target database to obtain the target historical interaction data that matches the semantic of the query information; in the case of obtaining at least one question and answer pair by analyzing the target historical interaction data, inserting the question and answer pair into a preset prompt template to obtain prompt information for processing the current query.

[0111] The process of converting information into a semantic vector representation in this embodiment has been described in the foregoing embodiments and will not be described in detail here. The query vector can be input into the vector retrieval subsystem of the target database to perform a similarity search and return the K most similar structured memory contents, which are usually successful QA pairs (question and answer pairs) in history. The prompt template is a string template with reserved example positions, for example, using string splicing or template rendering, such as, "Please refer to the following examples: Q: {q1} A: {a1}, now please answer: Q: {user_query} A:". In this way, a specific example and highly operational prompt word is dynamically generated, greatly improving the performance accuracy and reliability of large language models on specific tasks.

[0112] It should be noted that the related processing process in the data processing process of the target agent in the embodiments of the present application is similar to the agent training process, and can refer to the foregoing embodiments of the agent training, which will not be described in detail here.

[0113] Referring to Figure 3 which shows a flowchart of an agent training and use provided by an embodiment of the present application. The agent is mainly trained and applied based on context. Figure 3The function module of memory-enhanced context awareness shown in the middle is mainly to complete the collection of user interaction data with the agent (such as query, dialogue history, etc.), generate structured memory content through screening, compression, isolation and other processes, build target context memory, provide support for the agent to understand the task background and user demand, and make the agent callable associated memory to assist task execution. The function module of context learning is mainly to retrieve associated content to generate Few-shot (few-shot learning) examples according to the user-labeled question and answer pairs, dynamically splice them into prompt words, and enhance the agent's understanding and response ability to the task by means of historical question and answer experience, that is, use historical interaction to assist the logic of the current task. The function module of prompt word optimization is mainly to build task logic and set reward function, collect trajectory data through multiple rounds, train the model iteratively using reinforcement learning, and optimize the prompt word generation ability to make the agent output more in line with the task requirements. After the core work of the above three function modules is completed, the context of the agent is integrated and generated, which can be fed back to the agent usage link. The agent executes the task again based on the updated context, forms a "use-optimized context-reuse" cycle, continuously iterates and optimizes the performance of the agent in task execution, and realizes the progressive improvement of the agent's ability based on context engineering.

[0114] Referring to Figure 4 It shows a smart agent reinforcement learning training process diagram provided by an embodiment of the present application. In the task and data preparation stage, the agent and the task environment are defined first, the scene rules for the agent to execute the task are clarified, the training data is prepared to provide materials for model training, and then the initialized model is executed to build the basic framework for agent training.

[0115] In the training cycle execution stage, the agent behavior is sampled in turn in the training cycle to let the agent try to generate a trajectory in the task environment. Then the reward and loss are calculated to evaluate the pros and cons of the relevant strategy currently executed by the agent. In the process of back propagation and parameter update, the model parameter optimization strategy is adjusted according to the loss, and after one cycle, if the model convergence condition is not met, the training continues to be sampled until the convergence requirement is met. After convergence, the model is saved to retain the trained agent model, and finally the model effect is tested and the results are presented in a visual form through evaluation and visualization. The whole process optimizes the agent strategy through iterative optimization, and realizes the improvement of its ability in the task environment.

[0116] Referring to Figure 5 A large model context window construction process diagram based on multi-module input in an application scenario is provided by an embodiment of the present application. Taking the context window (Context Window) construction process of a large language model (LLM) in the scenario of the intelligent customer service responding to the user's query of "querying the order logistics progress this month" as an example for description.

[0117] In the prompt word optimization module, instruction content that fits the customer service task can be generated. For example, accurate prompt words are generated, and it is clearly required to reply to user logistics queries in a clear and concise style; Prepare "few-shot examples", such as past similar "query last month's order logistics" high-quality replies (user asks: When will the last month's order arrive? Customer service answers: Your last month's order number 12345 is currently in XX transfer station and is expected to arrive tomorrow morning); Supplement "other" information, such as limiting the reply to include the logistics estimated delivery time and the current site, as an instructional input to guide the LLM to perform the customer service task. In the context learning module, the content related to "order query" in the user's historical interaction is integrated with the context learning logic, and the output of the prompt word optimization module is input in the form of "instructions" to assist the LLM in understanding, such as associating with the user's previous "modify delivery address" interaction to let the LLM know the order-related information and respond more accurately to the logistics query. The memory-enhanced context awareness module can output two types of key information, such as knowledge-related information, which can be the extraction of basic data of the user's order (order time, product information, delivery address), logistics business rules (logistics query interface calling specification, different logistics state interpretation). The output information representing the work aspect can be to provide a callable logistics query API tool, an address verification tool, etc. These "knowledge" and "tools" are directly input into the context window to empower the model to perform tasks.

[0118] Finally, the outputs of various modules converge into the context window of the LLM, and the model calls the logistics query tool based on these information, combines the order knowledge with the customer service reply specification, and generates a reply like "Your last month's order number 67890, the product is XX, the current logistics is in XX station, and is expected to be delivered by 3pm the next day, and the delivery address is XX (the address you modified before) " to achieve multi-dimensional information collaborative driving of intelligent customer service to efficiently respond to user order queries.

[0119] In the embodiments of the present application, an intelligent agent training device is also provided, which is shown in Figure 6 The device comprises:

[0120] The acquisition unit 301 is configured to acquire target interaction data generated by the initial intelligent agent interacting with the target user;

[0121] The first updating unit 302 is configured to process the target interaction data and update a target database based on the processing result, wherein the target database is configured to store structured historical context information;

[0122] The first acquisition unit 303 is configured to acquire trajectory data generated by the initial intelligent agent performing a corresponding target task in a target task environment, wherein the trajectory data represents the complete process record data of the initial intelligent agent completing a complete target task;

[0123] The scoring unit 304 is configured to score the trajectory data based on a scoring model, to obtain a scoring result, wherein the scoring model is based on a user feedback signal extracted from the trajectory data.

[0124] The second updating unit 305 is configured to optimize the policy data of the initial agent based on the scoring result, to obtain a target agent, wherein the target agent is configured to perform the task based on the optimized policy data.

[0125] In a possible implementation, the first updating unit includes:

[0126] The extraction subunit is configured to extract target data in the target interaction data, if the target interaction data meets a condition of being converted into long-term memory, wherein the target data includes at least target user information, a target user intention, a processing flow performed by the initial agent in the interaction process, and a processing result generated by the initial agent based on the processing flow.

[0127] The packaging subunit is configured to package the target data in a structured format of data in a target database, to generate structured memory content.

[0128] The updating subunit is configured to update the target database based on the structured memory content.

[0129] In a possible implementation, the target database includes a key-value storage subsystem and a vector retrieval subsystem, wherein the updating subunit is configured to:

[0130] store the structured memory content in the key-value storage subsystem;

[0131] store a core text field in the structured memory content in the vector retrieval subsystem after the core text field is converted into a vector;

[0132] The processing procedure of retrieving the target historical context information corresponding to the target task environment in the target database includes:

[0133] calculating a vector representation of a core semantic feature of the target task environment;

[0134] retrieving, in the vector retrieval subsystem, a target vector corresponding to the vector representation of the core semantic feature and target structured memory content corresponding to the target vector;

[0135] retrieving, from the key-value storage subsystem, an interaction history record according to identification information recorded in the target structured memory content;

[0136] Fuse the interaction history record and the target structured memory content to obtain target historical context information.

[0137] In a possible implementation, the scoring unit includes:

[0138] The retrieval subunit is configured to retrieve historical context information related to the target task environment from the target database.

[0139] The result acquisition subunit is configured to input the historical context information, the completion condition corresponding to the target task, and the trajectory data into the scoring model to obtain a scoring result.

[0140] In a possible implementation, the second updating unit is specifically configured to:

[0141] According to the scoring result, determine a relative advantage and disadvantage order of each piece of trajectory data.

[0142] According to the relative advantage and disadvantage order, generate a reward signal corresponding to each piece of trajectory data; the reward signal is used to guide the optimization direction of the policy data of the model.

[0143] According to the reward signal, optimize the policy data of the initial agent to obtain a target agent, so that the target agent outputs trajectory data satisfying a high reward signal.

[0144] Specifically, the execution functions of each unit and subunit of the agent training apparatus can refer to the descriptions of the foregoing embodiments, which will not be described in detail here.

[0145] In another embodiment of the present application, a data processing apparatus is also provided, which can be seen from Figure 7 , and includes:

[0146] The receiving unit 401 is configured to receive query information of a user.

[0147] The retrieval unit 402 is configured to retrieve associated target historical interaction data from a target database according to the query information, the target database storing structured memory content updated iteratively based on interaction data in a target agent training process.

[0148] The generation unit 403 is configured to dynamically construct prompt information for processing the current query based on the target historical interaction data.

[0149] The second acquisition unit 404 is configured to input the query information and the prompt information into the target agent to obtain target output information corresponding to the query information.

[0150] In a possible implementation, the generation unit includes:

[0151] inserting a question and answer pair obtained by parsing the target historical interaction data into a prompt word template as a sample example, to obtain prompt information for processing the current query.

[0152] In a possible implementation, the inserting subunit is configured to:

[0153] calculate a semantic vector representation of the query information;

[0154] perform retrieval in the target database based on the semantic vector representation of the query information, to obtain the target historical interaction data that is semantically matched with the query information;

[0155] insert the question and answer pair obtained by parsing the target historical interaction data into a preset prompt word template, to obtain prompt information for processing the current query.

[0156] The execution processes of the various units of the data processing apparatus corresponding to the embodiments of the present application can be referred to the descriptions of the foregoing embodiments, which will not be described in detail here.

[0157] In addition, it should be noted that the apparatus embodiments described above are merely schematic, wherein the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. According to actual needs, part or all of the modules can be selected to achieve the purpose of the embodiments of the present application. In addition, in the apparatus embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which can be implemented as one or more communication buses or signal lines.

[0158] Those skilled in the art can clearly understand that the application can be implemented by means of software plus necessary universal hardware, and of course can also be implemented by means of dedicated hardware including special-purpose integrated circuits, special-purpose CPUs, special-purpose memories, special-purpose components, etc. Generally, any function completed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure for implementing the same function can also be various, such as analog circuits, digital circuits, or special-purpose circuits, etc. However, for the present application, software program implementation is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a readable storage medium, such as a floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk, etc., and includes a plurality of instructions for causing a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in various embodiments of the present application.

[0159] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product.

[0160] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another, for example, the computer instructions can be transmitted from one website, computer, training device or data center to another website, computer, training device or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be stored by a computer or a data storage device such as a training device, a data center, etc. integrated with one or more available media. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

[0161] The various embodiments described in this specification are implemented in the context of a progressive manner, each embodiment focusing on the differences from other embodiments, and the same or similar parts between embodiments can be mutually referred to. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part description. The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to the embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An agent training method, characterized by, The method comprises: collecting target interaction data generated by an initial agent interacting with a target user; processing the target interaction data and updating a target database based on the processing result, the target database being configured to store structured historical context information; obtaining trajectory data generated by the initial agent in a target task environment when performing a corresponding target task, the trajectory data representing a complete process record of the initial agent completing a target task; scoring the trajectory data based on a scoring model to obtain a scoring result, the scoring model being evaluated based on user feedback signals extracted from the trajectory data; optimizing strategy data of the initial agent based on the scoring result to obtain a target agent, the target agent being configured to perform a task based on the optimized strategy data.

2. The method of claim 1, wherein, The processing of the target interaction data and the updating of the target database based on the processing result comprises: extracting target data in the target interaction data under the condition that the target interaction data meets the condition of being converted into long-term memory, the target data at least including target user information, target user intent, a processing flow performed by the initial agent in the interaction process, and a processing result generated by the initial agent based on the processing flow; packaging the target data in a structured format of data in the target database to generate structured memory content; updating the target database based on the structured memory content.

3. The method of claim 1, wherein, The target database comprises a key-value storage subsystem and a vector retrieval subsystem, and the updating of the target database based on the structured memory content comprises: storing the structured memory content in the key-value storage subsystem; after converting a core text field in the structured memory content into a vector, storing the vector in the vector retrieval subsystem; the process of retrieving target historical context information corresponding to the target task environment in the target database comprises: calculating a vector representation of a core semantic feature of the target task environment; in the vector retrieval subsystem, retrieving a target vector corresponding to the vector representation of the core semantic feature and target structured memory content corresponding to the target vector; according to identification information recorded in the target structured memory content, retrieving an interaction history record from the key-value storage subsystem; fusing the interaction history record and the target structured memory content to obtain target historical context information.

4. The method of claim 1, wherein, The scoring of the trajectory data based on the scoring model to obtain a scoring result comprises: retrieving historical context information related to the target task environment from the target database; inputting the historical context information, a completion condition corresponding to the target task, and the trajectory data into the scoring model to obtain a scoring result.

5. The method of claim 1, wherein, The optimization of the strategy data of the initial agent based on the scoring result to obtain a target agent comprises: determining a relative ranking of each piece of trajectory data according to the scoring result; According to the relative advantage and disadvantage ranking, a reward signal corresponding to each piece of trajectory data is generated; the reward signal is used to guide the optimization direction of the policy data of the model; According to the reward signal, the policy data of the initial agent is optimized to obtain a target agent, so that the target agent outputs trajectory data satisfying a high reward signal.

6. A data processing method, characterized by, The target agent obtained by training according to any one of claims 1-5 is applied to, comprising: receiving query information of a user; According to the query information, the associated target historical interaction data is retrieved in the target database, and the target database stores the structured memory content updated iteratively based on the interaction data in the training process of the target agent; Based on the target historical interaction data, prompt information for processing the current query is dynamically constructed; The query information and the prompt information are input into the target agent to obtain target output information corresponding to the query information.

7. The method of claim 6, wherein, The target historical interaction data is based on, and the prompt information for processing the current query is dynamically constructed, comprising: In the case of obtaining at least one question and answer pair from the target historical interaction data, the question and answer pair is inserted into the prompt word template as a sample example to obtain the prompt information for processing the current query.

8. The method of claim 7, wherein, The question and answer pair is inserted into the prompt word template as a sample example to obtain the prompt information for processing the current query, comprising: Calculate the semantic vector representation of the query information; Based on the semantic vector representation of the query information, the target historical interaction data that matches the semantics of the query information is retrieved in the target database; In the case of obtaining at least one question and answer pair from the target historical interaction data, the question and answer pair is inserted into the preset prompt word template to obtain the prompt information for processing the current query.

9. An agent training apparatus characterized by comprising: Comprising: The acquisition unit is used for collecting target interaction data generated by the initial agent interacting with the target user; The first update unit is used for processing the target interaction data and updating the target database based on the processing result, and the target database is used for storing structured historical context information; The first acquisition unit is used for acquiring trajectory data generated by the initial agent in a target task environment when performing a corresponding target task, and the trajectory data represents the complete process record data of the initial agent completing a complete target task; The scoring unit is used for scoring the trajectory data based on a scoring model to obtain a scoring result, and the scoring model is evaluated based on the user feedback signal extracted from the trajectory data; The second update unit is used for optimizing the policy data of the initial agent based on the scoring result to obtain a target agent, and the target agent is configured to perform a task based on the optimized policy data.

10. A data processing apparatus, characterized by, Comprising: The receiving unit is used for receiving query information of a user; The retrieval unit is used for retrieving associated target historical interaction data in a target database according to the query information, and the target database stores structured memory content updated iteratively based on interaction data in the training process of the target agent; The generating unit is configured to dynamically construct prompt information for processing the current query based on target historical interaction data. The second obtaining unit is configured to input the query information and the prompt information into the target agent, and obtain target output information corresponding to the query information.

Images