A multi-agent reinforcement learning method and device supporting course learning

Abstract

This invention discloses a multi-agent reinforcement learning method and apparatus to support course learning. The method includes: assigning training tasks from a training task set to each agent to be trained to generate task requests; forwarding the task requests to a large language model via an intermediate service component that responds to address variables within the task requests; receiving the execution data returned by the large language model and converting it, along with the task requests, into a framework-readable format; generating target return data and calculating task reward values; adjusting the model weights of the associated agents to be trained; adjusting the difficulty of the training task set based on the reward values ​​of each task to obtain a new training task set; and iterating until the number of rounds of assignment reaches an assignment threshold or the number of tasks in the training task set is less than a batch threshold. Thus, by adding an intermediate service component and updating the difficulty of the task training set, the agents can directly interact with the external environment while focusing on tasks of varying difficulty, effectively improving the training efficiency of the agents.

Description

Technical Field

[0001] This invention relates to the field of multi-agent reinforcement learning technology, and in particular to a multi-agent reinforcement learning method and apparatus that supports curriculum learning. Background Technology

[0002] With the widespread application of Large Language Model-based Agents (LLM-based Agents) in complex business scenarios such as data analysis and process automation, how to continuously optimize the agent's decision-making capabilities through reinforcement learning (RL) has become a key technological direction. AgentRL (Agent Reinforcement Learning) methods iteratively optimize agent policies through environmental interaction and reward feedback, and are particularly suitable for scenarios requiring multi-step reasoning and tool invocation, such as Chat Business Intelligence (ChatBI), aiming to improve the accuracy and adaptability of agents in real business environments.

[0003] To address the challenge of inconsistencies between training and real online environments, existing solutions typically employ methods based on dedicated frameworks such as AGL (Agent Guidance Language) to construct simulated training environments. This involves reimplementing agent logic, tool interfaces (such as MCP services), and prompt word templates in the training system to replicate the real interaction process as closely as possible.

[0004] However, the training process of this scheme is isolated from the real business system. The interaction objects of the intelligent agent are simulated service interfaces rather than the general intelligent agents actually deployed. This causes the training environment and process to deviate from the continuous evolution requirements of the real business scenario, limiting the optimization efficiency and performance ceiling of the intelligent agent. Summary of the Invention

[0005] This invention provides a multi-agent reinforcement learning method and apparatus that supports course learning. It solves the technical problem that the training process of existing solutions is isolated from the real business system, and the interaction objects of the agents are simulated service interfaces rather than the actual deployed general agents. This causes the training environment and process to deviate from the continuous evolution requirements of the real business scenario, which limits the optimization efficiency and performance limit of the agents.

[0006] This invention provides a multi-agent reinforcement learning method for supporting course learning, applied to an agent training framework. The agent training framework communicates and connects with intermediate service components and multiple agents to be trained. The method includes:

[0007] When a training task set is received, each training task in the training task set is assigned to each of the agents to be trained, and the agents to be trained generate corresponding task requests.

[0008] The intermediate service component responds to any of the task requests and forwards the task request to the corresponding large language model according to the address variable in the task request.

[0009] The intermediate service component receives the execution data returned by the large language model and converts it into a framework-readable format along with the task request to generate the target return data.

[0010] Calculate the task reward value corresponding to each of the target return data, and adjust the model weights of the agent to be trained associated with the target return data;

[0011] The difficulty of the training task set is adjusted according to the reward value of each task to obtain a new training task set.

[0012] Jump to execute the step of assigning each training task in the training task set to the agent to be trained one by one, until the number of allocation rounds reaches a preset allocation threshold or the number of tasks in the training task set is less than a preset batch threshold, and the agent to be trained at the current moment is determined as the target agent.

[0013] Optionally, it also includes:

[0014] Send the set of model addresses corresponding to each training task to the intermediate service component;

[0015] When the intermediate service component receives any of the task requests, it matches the dynamic interface address corresponding to the address variable from the model address set;

[0016] The intermediate service component updates the current component's output interface address to a dynamic interface address.

[0017] The task request is forwarded to the large language model corresponding to the output interface address of the current component.

[0018] Optionally, the method further includes a large model proxy component, which is associated with multiple large language models; the large model proxy component is communicatively connected to the agent training framework and the intermediate service component, respectively; the method further includes:

[0019] The large model proxy component responds to the task request sent by the intermediate service component and sends the task request to the corresponding large language model according to the dynamic interface address or the address variable.

[0020] When the large model proxy component receives execution data from the large language model, it returns the execution data to the intermediate service component.

[0021] Optionally, the step of assigning each training task in the training task set to each of the agents to be trained one by one when a training task set is received, and generating a corresponding task request through the agent to be trained, includes:

[0022] When a training task set is received, the training tasks in the training task set are reorganized according to a preset reorganization logic to obtain a new training task set.

[0023] Each training task in the training task set is assigned to each of the intelligent agents to be trained;

[0024] The training task is structured and loaded into a structured template by the agent to be trained, thereby generating a task request corresponding to the training task.

[0025] Optionally, the step of responding to any of the task requests through the intermediate service component and forwarding the task request to the corresponding large language model according to the address variable in the task request includes:

[0026] The intermediate service component responds to any of the task requests and extracts the payload within the task request.

[0027] The intermediate service component retrieves the address variable matching the variable name from within the payload;

[0028] The intermediate service component updates the current component's output interface address to an address variable.

[0029] The intermediate service component forwards the task request to the large language model corresponding to the output interface address of the current component.

[0030] Optionally, the step of calculating the task reward value corresponding to each of the target return data and adjusting the model weights of the agent to be trained associated with the target return data includes:

[0031] At least one reward calculation model is invoked to calculate the data gap between the target return data and the actual return data;

[0032] Based on the data gap and the preset reward calculation rules, calculate the task reward value for each of the target returned data;

[0033] A preset group-relative strategy optimization algorithm is used to adjust the model weights of the agent to be trained associated with the target return data according to the reward values ​​of each task.

[0034] Optionally, the step of adjusting the difficulty of the training task set according to the reward value of each task to obtain a new training task set includes:

[0035] The task difficulty of the training task corresponding to each task reward value is determined according to the numerical range in which each task reward value is located.

[0036] The training tasks are clustered according to their difficulty to obtain multiple task groups;

[0037] The training tasks within each task group are reorganized according to a preset reorganization logic to obtain multiple updated task groups.

[0038] Select tasks to be expanded from the updated task group whose task difficulty meets the first preset difficulty, and expand the tasks to be expanded according to the preset expansion rules to obtain an expanded task group containing multiple expanded tasks.

[0039] Select a first task from the updated task group whose task difficulty meets the second preset difficulty, and construct a first task subset;

[0040] A second task subset is constructed by selecting a second proportion of the expanded task group and combining it with all the updated task groups whose difficulty meets the first preset difficulty; the ratio between the first task subset and the second task subset is a preset construction ratio.

[0041] A new training task set is generated by using the first task subset and the second task subset.

[0042] Optionally, it also includes:

[0043] If the average reward value of any of the training tasks is lower than a preset reward threshold for multiple consecutive allocation rounds, then the training task is removed.

[0044] Optionally, after performing the step of receiving the execution data returned by the large language model through the intermediate service component, the method further includes:

[0045] The execution data is returned to the agent to be trained via the intermediate service component.

[0046] The training agent performs logical analysis on the execution data and determines whether there is a need for re-calling based on the analysis results.

[0047] If so, a new task request is generated according to the analysis results and sent to the intermediate service component. Then, the process jumps to the step of responding to any of the task requests through the intermediate service component and forwarding the task request to the corresponding large language model according to the address variable in the task request.

[0048] If not, then all the execution data is integrated and sent to the intermediate service component.

[0049] This invention also provides a multi-agent reinforcement learning method apparatus to support course learning, applied to an agent training framework, wherein the agent training framework is interconnected with an intermediate service component and multiple agents to be trained, and the apparatus includes:

[0050] The task request generation module is used to, when a training task set is received, allocate each training task in the training task set to each of the intelligent agents to be trained, and generate corresponding task requests through the intelligent agents to be trained.

[0051] The request forwarding module is used to respond to any of the task requests through the intermediate service component and forward the task request to the corresponding large language model according to the address variable in the task request;

[0052] The data return module is used to receive the execution data returned by the large language model through the intermediate service component and convert it into a framework-readable format together with the task request to generate the target return data;

[0053] The model weight adjustment module is used to calculate the task reward value corresponding to each of the target return data and adjust the model weights of the agent to be trained associated with the target return data.

[0054] The task set update module is used to adjust the difficulty of the training task set according to the reward value of each task to obtain a new training task set;

[0055] The agent loop training module is used to jump to execute the step of assigning each training task in the training task set to the agent to be trained one by one until the number of allocation rounds reaches a preset allocation threshold or the number of tasks in the training task set is less than a preset batch threshold, and then determine the agent to be trained at the current moment as the target agent.

[0056] As can be seen from the above technical solutions, the present invention has the following advantages:

[0057] Upon receiving a training task set, the system assigns each training task to a trainable agent, which then generates a task request. The intermediate service component responds to the address variable within the task request, forwarding the request to the large language model. It receives the execution data returned by the large language model and converts it, along with the task request, into a framework-readable format, generating target return data. The system calculates the task reward value for each target return data point and adjusts the model weights of the trainable agents associated with it. Based on the task reward values, the difficulty of the training task set is adjusted to obtain a new training task set. Training continues cyclically until the number of allocation rounds reaches the allocation threshold or the number of tasks in the training task set is less than the batch threshold. Thus, through the addition of the intermediate service component and the updating of the task training set difficulty, the agent can directly interact with the external environment while focusing on tasks of varying difficulty, effectively improving the agent's training efficiency. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 A flowchart illustrating the steps of a multi-agent reinforcement learning method for supporting course learning, provided in an embodiment of the present invention;

[0060] Figure 2 This is a schematic diagram illustrating the architecture between the agent training framework and other components in an embodiment of the present invention.

[0061] Figure 3 This is a structural block diagram of a multi-agent reinforcement learning method device that supports course learning, provided in an embodiment of the present invention. Detailed Implementation

[0062] This invention provides a multi-agent reinforcement learning method and apparatus to support course learning, which addresses the technical problem that existing solutions isolate the training process from real business systems, and the interaction objects of the agents are simulated service interfaces rather than actually deployed general agents. This leads to the training environment and process deviating from the continuous evolution requirements of real business scenarios, limiting the optimization efficiency and performance limits of the agents.

[0063] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0064] Please see Figure 1 , Figure 1 The flowchart illustrates the steps of a multi-agent reinforcement learning method for supporting course learning, as provided in an embodiment of the present invention.

[0065] This invention provides a multi-agent reinforcement learning method to support course learning, applied to an agent training framework. The agent training framework, intermediate service components, and multiple agents to be trained communicate and connect with each other. The method includes:

[0066] Step 101: When the training task set is received, each training task in the training task set is assigned to each agent to be trained, and the agent to be trained generates the corresponding task request.

[0067] The agent training framework refers to a framework, such as the AGL agent training framework, used to adjust the model weights of agents.

[0068] The training task set refers to the set of tasks used to carry out reinforcement learning training on the agent to be trained. These include, but are not limited to, various tasks such as data analysis, data query, visualization generation, and report creation in ChatBI or other different business scenarios.

[0069] An agent to be trained refers to an agent that has not completed reinforcement learning optimization and needs to adjust the model weights through training tasks. It can be a single agent or multiple agents. Multiple agents can include, but are not limited to, a coordinator agent and various professional data service agents.

[0070] A task request refers to a request data packet generated after the agent to be trained parses the training task. It is used to send a task execution request to the large language model and contains core information such as address variables, task description, and running parameters.

[0071] In this embodiment of the invention, when the agent training framework receives an externally input training task set, it first performs random shuffling on the training task set, and then, based on the ChatBI business function division and business adaptability of each agent to be trained, it matches and assigns the training tasks one by one to the corresponding agents to be trained. Each agent to be trained parses the business requirements of the training task based on its own functional attributes, and completes the parsing and standardized encapsulation of the task in combination with the preset Agent Prompt template, generating a task request containing address variables, task description, and agent running parameters, wherein the address variables are dynamic API interface addresses pointing to the corresponding large language model.

[0072] In addition, training tasks can be personalized by combining the current training progress and functional strengths of each agent to be trained. Tasks in weak business areas of the agent can be allocated in a tiered manner, while the proportion of tasks in the strength areas can be increased in a single allocation. At the same time, a unique agent identifier and task identifier are added to each task request to achieve traceability of the entire task execution process, which is convenient for the location and analysis of training problems in the future.

[0073] Please see Figure 2 , Figure 2 This is a schematic diagram illustrating the architecture between the agent training framework and other components in an embodiment of the present invention.

[0074] In this embodiment, the architecture is a layered collaborative system supporting reinforcement learning training, with its core comprising four major modules: an agent training framework, agents to be trained, intermediate service components, and a large language model cluster. The agent training framework, as the global control core, not only issues tasks to the agents to be trained and receives model weight adjustment feedback, but also receives target return data from the intermediate service components, and achieves self-iterative optimization through updates to the training task set. The agents to be trained are the main training execution entities; after receiving tasks, they generate task requests and complete computational power interaction with the large language model cluster via the intermediate service components. The intermediate service components, acting as communication hubs, are responsible for forwarding task requests, transmitting execution data back, and format conversion, decoupling the agents from the large model. The large language model cluster (models 1 to n), as the computational power support layer, only receives tasks and returns results through the intermediate service components, and is not directly coupled with other modules, ensuring the system's scalability and heterogeneous adaptability.

[0075] The overall data flow forms a closed loop of forward task execution and reverse optimization iteration. In the forward process, the agent to be trained generates a task request, which is forwarded to the target large language model via an intermediate service component. The large language model performs calculations and outputs execution data, which is then returned via the intermediate service component. In the reverse process, the intermediate service component converts the execution data and task request into a framework-readable format, obtains the target return data, and submits it to the agent training framework. The framework evaluates the reward value based on the target return data and then issues a model weight adjustment instruction to the agent to be trained, driving the agent to complete policy iteration. At the same time, the framework triggers the update of the training task set based on the training effect, dynamically adjusting the task difficulty and composition to make subsequent training tasks more aligned with the agent's capabilities, forming a complete closed loop of course-learning training data flow.

[0076] In one example of the present invention, step 101 may include the following sub-steps:

[0077] When a training task set is received, the training tasks in the training task set are reorganized according to the preset reorganization logic to obtain a new training task set.

[0078] Each training task in the training task set is assigned to each agent to be trained.

[0079] The training task is structured and loaded into a structured template by the agent to be trained, generating the task request corresponding to the training task.

[0080] Structured templates refer to standardized task request generation templates adapted to ChatBI business scenarios. They support variable input and include fixed core fields such as address variables, task descriptions, and business operation parameters.

[0081] In this embodiment, after receiving a training task set based on business needs, the training tasks within the set are reorganized according to preset reorganization logic, such as task order rearrangement, difficulty level division rearrangement, or random arrangement, to obtain a new training task set adapted to the learning pace of the course. This achieves a gradient arrangement of training tasks and eliminates invalid and noisy tasks, conforming to the training pattern of learning from easy to difficult, avoiding meaningless and ineffective training by the agent, and effectively improving the training efficiency and model convergence speed of the agent's reinforcement learning. Then, according to the order of training tasks in the new training task set or according to the business function division or training requirements of each agent to be trained, each training task in the new training task set is matched and assigned to the corresponding agent to be trained one by one. Each agent to be trained performs semantic decomposition, core business requirement extraction, and task attribute identification structured parsing on the received training task. The key information obtained from the parsing is loaded into a preset structured template that supports variable input according to the field mapping relationship. Core fields such as address variables, task description, and business operation parameters are automatically filled in. Finally, a standardized task request that can be recognized and forwarded by the intermediate service component is generated. By standardizing and normalizing the generation of task requests, the format differences of requests generated by different agents are eliminated, which meets the parsing and forwarding requirements of the intermediate service component, improves the smoothness of request flow, makes the training process closer to the actual online operating environment, and ensures the stability of the effect after the agent is trained and put into operation.

[0082] Furthermore, for the structured task request generation stage, knowledge graphs can be integrated into the structured parsing process. This allows for matching similar business scenarios using the knowledge graph, assisting the agent in extracting the core requirements of novel and complex training tasks, and improving the agent's ability to parse unknown and complex tasks. Additionally, dynamic fields can be added to the structured templates to load specific business fields based on different business scenario types, such as data querying, data visualization generation, and enterprise data report creation.

[0083] In traditional training schemes, to achieve fine-tuning of the agent to be trained, the agent can be directly modified. However, this requires developing a separate interface to support dynamic output, which affects its actual stable operation. Furthermore, it necessitates development for different engineering platforms, which is time-consuming and labor-intensive, and cannot be directly integrated with any existing large language models or agents for training. Therefore, this invention provides an intermediate service component that links the AGL agent training framework with the agent to be trained on the engineering side, enabling end-to-end model fine-tuning.

[0084] Step 102: Respond to any task request through the intermediate service component, and forward the task request to the corresponding large language model according to the address variable in the task request;

[0085] Middleware components refer to middleware services that enable the interaction between the AGL agent training framework and the engineering-side agents. They possess core capabilities such as fixed entry addresses, dynamic exit addresses, and streaming / non-streaming request forwarding. Figure 2 The TLLM proxy component in [the context of the application].

[0086] Address variables refer to dynamic API interface address identifiers carried in task requests that point to the large language model. They can be parsed and extracted from the task request Prompt to match a unique large language model interface for each training task.

[0087] Large language models refer to the basic models that provide core capabilities such as natural language processing, data analysis, and logical computation for the intelligent agent to be trained. They are the core computing power support for the intelligent agent to perform training tasks.

[0088] In this embodiment, the intermediate service component uses a fixed API interface as the request entry point to receive task requests sent by each agent to be trained in real time. It performs structured parsing on any received task request, extracts the address variables, and performs validity verification. After successful verification, it updates its current component output interface address according to the dynamic API interface address of the large language model corresponding to the address variable. The task request is then forwarded to the corresponding large language model in a streaming or non-streaming manner through the current component output interface address. During the forwarding process, the original business information and agent running parameters of the task request are kept intact and are not lost or tampered with.

[0089] In addition, request splitting and load balancing mechanisms can be added to the intermediate service components. When multiple task requests point to the same large language model, requests are queued and forwarded in batches according to the real-time computing power load of the large language model to avoid overloading the computing power of a single large language model. At the same time, an address variable fault tolerance mechanism is added to automatically identify invalid address variables and match the optimal backup large language model address to ensure the continuity and stability of task request forwarding.

[0090] In one example of the present invention, step 102 may include the following sub-steps:

[0091] The intermediate service component responds to any task request and extracts the payload within the task request.

[0092] The address variable matching the variable name is retrieved from the payload through the intermediate service component;

[0093] The intermediate service component updates the current component's output interface address to the address variable.

[0094] The intermediate service component forwards the task request to the large language model corresponding to the output interface address of the current component.

[0095] The payload refers to the data portion of the task request data packet that carries the core business logic and execution parameters. It excludes the encapsulation information from the transport layer and is the core object for data parsing by the intermediate service components.

[0096] Address variables refer to dynamic parameters embedded in the payload that are used to identify the location of the target large language model network. They are associated with specific model interface addresses using preset variable names as retrieval identifiers.

[0097] The current component output interface address refers to the outgoing network address of the intermediate service component when it initiates a request to the outside world at the current moment. It serves as the target address for forwarding task requests and supports real-time dynamic updates based on the address variable.

[0098] In this embodiment, the intermediate service component responds in real time to task requests sent by any agent to be trained through a preset ingress listening port. Upon establishing a communication connection, it immediately decapsulates the request data packet, stripping away redundant transport layer information to extract the payload, thus reducing the amount of data parsed by the intermediate service component. Subsequently, the intermediate service component searches and matches the structured data of the payload according to preset variable names, extracting the corresponding address variables. After obtaining the address variables, the intermediate service component triggers an interface address update mechanism, dynamically refreshing its current component output interface address from the default state to the network address pointed to by the retrieved address variable. This frees the intermediate service component from the limitation of a fixed exit point, enabling flexible adaptation to heterogeneous deployment environments with multiple language models. After completing the address update, the intermediate service component maintains the original business data of the payload without tampering and forwards the standardized task request completely to the corresponding large language model according to the updated component output interface address. For example, taking the intermediate service component TLLM Proxy as an example, after receiving a task request and extracting the payload, it extracts prompt / messages and parameters such as temperature and top_p from the payload. Since prompts typically support variable input (i.e., the prompt template contains variable parameters such as `{{current_time}}` and `{{sandbox_file}}`), a special variable, such as `{{conversation_link}}`, can be set. Each time a task requests the API of the agent to be trained, this `{{conversation_link}}` (i.e., `llm_endpoint`) is included. In this case, the TLLM Proxy can extract the `llm_endpoint` from the task request, effectively avoiding additional data request flows and enabling multi-threaded task request forwarding.

[0099] In addition, primary and backup variable names can be set for address variables. When the primary variable name fails to be retrieved, the backup variable name will be automatically matched. At the same time, priority can be set for address variables from different sources to resolve multi-variable conflict issues and further improve the robustness of address matching.

[0100] Optionally, before updating the current component's output interface address, a lightweight liveness detection request can be used to verify whether the large language model corresponding to the address variable is online and whether the computing power is sufficient. If the address is invalid, the backup model address list will be automatically called to avoid forwarding the request to an unavailable node and improve the success rate of forwarding.

[0101] In one example of the present invention, the method further includes the following steps:

[0102] Send the set of model addresses corresponding to each training task to the intermediate service component;

[0103] When the intermediate service component receives any task request, it matches the dynamic interface address corresponding to the address variable from the model address set.

[0104] The intermediate service component updates the current component's output interface address to a dynamic interface address;

[0105] Forward the task request to the large language model corresponding to the output interface address of the current component.

[0106] The model address set refers to the configuration set that corresponds to the training task and the network address of the large language model. It uses the address variable as the index key to associate one or more valid large language model access addresses.

[0107] In this embodiment, the agent training framework actively pushes the model address set corresponding to each training task to the intermediate service component in a structured configuration. The intermediate service component receives the model address set and completes local caching and index construction. When the intermediate service component listens for and receives a task request submitted by any agent to be trained, it first parses the address variable carried in the request, then uses the local index to perform precise matching in the model address set to obtain the dynamic interface address mapped by the address variable. Subsequently, the intermediate service component updates its current component output interface address to the matched dynamic interface address. After completing the address update, the intermediate service component forwards the unmodified original task request to the corresponding large language model via network call according to the updated component output interface address. Thus, by matching the dynamic interface address in the set based on the address variable, a flexible association between tasks and models is established. This allows the system to adapt to model expansion, node migration, or version iteration without modifying the core code of the agent or middleware; only the address set needs to be updated. This greatly improves the system's maintenance efficiency and scalability.

[0108] In one example of the present invention, a large model proxy component is also included, which is associated with multiple large language models; the large model proxy component is communicatively connected to the agent training framework and the intermediate service component respectively; the method further includes the following steps:

[0109] The large model proxy component responds to task requests sent by the intermediate service component and sends the task requests to the corresponding large language model according to the dynamic interface address or address variable.

[0110] When the large model proxy component receives execution data from the large language model, it returns the execution data to the intermediate service component.

[0111] The large model proxy component refers to a unified proxy service for large language models built on a standardized proxy framework. It associates multiple large language models of different types and deployment methods, and serves as a communication hub between the intermediate service component and the large language model. It undertakes the core functions of request forwarding, data format adaptation, and unified model management, and achieves bidirectional communication with the agent training framework and the intermediate service component.

[0112] Execution data refers to the dataset returned by the large language model after it receives and executes the training task request, which includes business processing results, data analysis process, and logical calculation conclusions.

[0113] In this embodiment, the large model proxy component can be set up within the agent training framework or set up independently. After receiving a task request forwarded by the intermediate service component through a preset communication interface, it parses the request to obtain the dynamic interface address or address variable. It then performs accurate matching of the target large language model within its associated large language model resource pool. Subsequently, it adapts the task request to a lightweight format according to the interface interaction specifications of the target large language model, ensuring that the request can be recognized and executed by the target model. The adapted task request is then forwarded to the corresponding large language model. When the large language model completes the computation and execution of the training task and returns the execution data, the large model proxy component receives the execution data, performs unified format standardization processing to eliminate format differences in the data returned by different large language models, and then returns the processed execution data to the intermediate service component that initiated the request along the original communication link. Thus, by setting up the large model proxy component, the direct association between the intermediate service component and the large language model is decoupled. This eliminates the need for the intermediate service component to adapt to the heterogeneous interface specifications of different large language models, significantly reducing the development and maintenance costs of the system. Simultaneously, it allows the intermediate service component to focus on the core functions of address updating and request forwarding. Meanwhile, by associating the large model agent component with multiple large language models, a unified model management entry point is provided for the system. This allows the agent training framework to centrally monitor and schedule multiple models through this component, thereby improving the system's compatibility and scalability with large language models.

[0114] In addition, the large model proxy component can monitor the running status of each associated large language model in real time through lightweight liveness detection requests. When a target model node is detected to be faulty, unresponsive, or lacking computing power, it automatically matches an available model from the backup model list and forwards the request without manual intervention, ensuring the continuity of training task execution.

[0115] Step 103: Receive the execution data returned by the large language model through the intermediate service component and convert it into a framework-readable format along with the task request to generate the target return data;

[0116] Execution data refers to the dataset returned by the large language model after receiving and executing task requests, which includes business processing results, data analysis process, and logical calculation conclusions;

[0117] A framework-readable format refers to a structured data format that conforms to the data parsing specifications of an agent training framework, and can be directly recognized by the framework and used for subsequent reward calculations and model weight adjustments.

[0118] In this embodiment, the intermediate service component listens to and receives the execution data returned by the large language model for each task request in real time. It associates and binds the execution data with the corresponding original task request one by one. According to the data parsing and format conversion rules preset by the AGL agent training framework, it performs field mapping, structure reorganization, and data type adaptation processing on the associated dataset, converting the unstructured execution data into a standardized framework-readable format. Finally, it generates target return data containing basic task information, large language model execution results, and a full record of interface calls, and pushes it to the AGL agent training framework in real time.

[0119] In addition, timestamps and data version identifiers can be added to the target returned data, which facilitates the AGL framework to trace training data, manage versions, and review and analyze training results.

[0120] After performing the step of receiving the execution data returned by the large language model through the intermediate service component, the method further includes the following steps:

[0121] The execution data is returned to the agent to be trained via an intermediate service component;

[0122] The agent to be trained performs logical analysis on the execution data, and determines whether there is a need for re-calling based on the analysis results.

[0123] If so, a new task request is generated according to the analysis results and sent to the intermediate service component. The process then jumps to the steps of responding to any task request through the intermediate service component and forwarding the task request to the corresponding large language model according to the address variable in the task request.

[0124] If not, then all execution data is integrated and sent to the intermediate service component.

[0125] Logical analysis refers to the core operations performed by the agent to be trained on the execution data returned by the large language model, such as ChatBI business logic verification, result integrity verification, and task completion evaluation.

[0126] In this embodiment, after receiving the execution data returned by the large language model and completing the format standardization process, the intermediate service component returns the execution data to the agent under training that initiated the task request via a preset communication link. When the agent under training receives the execution data, it performs a comprehensive verification and evaluation of the completeness of the business results, logical consistency, and task completion rate of the execution data. Based on preset task completion judgment rules, it determines whether there is a need to call the large language model again according to the logical analysis results. If a need for a re-call exists, the agent under training generates a new task request containing new address variables and business instructions based on the supplementary processing points identified in the logical analysis. It sends the new task request to the intermediate service component and triggers a process jump, re-executing the steps of the intermediate service component responding to the task request and forwarding it to the corresponding large language model according to the address variables. If no need for a re-call exists, the agent under training structurally integrates all the execution data obtained in this training task processing flow, generates an integrated dataset, and sends it to the intermediate service component, completing the execution data processing for this round.

[0127] In addition, a dynamically adjustable maximum number of re-calls can be set for a single training task to avoid training process lag caused by infinite repeated calls. When the task requirements are not met even after reaching the threshold, the training task is automatically marked as an abnormal task and fed back to the agent training framework, which facilitates subsequent intervention and analysis by the developers.

[0128] Step 104: Calculate the task reward value corresponding to the data returned by each target, and adjust the model weights of the agent to be trained associated with the data returned by the target.

[0129] The task reward value refers to the evaluation value obtained by calculating the matching degree between the execution result in the target returned data and the real answer based on the large model evaluation method. It is the core indicator for measuring the effect of the agent to be trained in performing training tasks.

[0130] Model weights refer to the core model parameters that determine the business execution capabilities of the agent being trained.

[0131] In this embodiment, after the agent training framework receives the target return data, it compares the execution results in each target return data with the corresponding real return data in multiple dimensions, calculates the task reward value corresponding to each target return data according to the preset reward calculation rules, and then uses the group relative policy optimization algorithm to use the input and output data of the large language model call and the task reward value as the actual training samples. Through high-level credit allocation, the final reward value of the entire task is allocated to each large language model call action in the trajectory. Then, through low-level policy update, the model weights of the agent to be trained associated with the target return data are optimized and adjusted in a targeted manner.

[0132] In this embodiment, in addition to the matching degree of core business data, evaluation dimensions such as task execution efficiency, data analysis logic rigor, and result presentation professionalism are added. Differentiated weight coefficients are set for each dimension, and a comprehensive task reward value is calculated to make the reward value more in line with actual business needs. In addition, during the model weight adjustment stage, a hierarchical weight adjustment strategy can be adopted, setting different weight adjustment ranges for the core functional modules and auxiliary functional modules of the agent to be trained. At the same time, the effect of weight adjustment is predicted by combining historical training data to avoid overfitting or underfitting problems in the agent model.

[0133] In one example of the present invention, step 104 may include the following sub-steps:

[0134] Call at least one reward calculation model to calculate the data gap between the target return data and the actual return data;

[0135] Calculate the task reward value for each target's returned data based on the data gap and the preset reward calculation rules;

[0136] A pre-defined group-relative strategy optimization algorithm is used to adjust the model weights of the agent to be trained in relation to the target return data according to the reward value of each task.

[0137] The reward calculation model refers to a difference quantification evaluation model adapted to business scenarios, which can quantify and analyze the difference between the target returned data and the actual returned data from multiple dimensions such as data integrity, logical consistency, and business result fit.

[0138] Data gap refers to a quantitative indicator output by the reward calculation model that represents the degree of difference between the target returned data and the actual returned data, covering numerical deviation, logical deviation, business dimension deviation, etc.

[0139] The reward calculation rule refers to the mapping rule based on the concepts of course learning and reinforcement learning, which establishes an inverse correlation between data gap and task reward value (the smaller the data gap, the higher the task reward value).

[0140] Group Relative Policy Optimization (GRPO) is a policy gradient algorithm for optimizing multi-agent collaborative training scenarios. It updates the policy by comparing the relative reward values ​​among agents within a group, thus balancing the performance improvement of single agents with the optimization of multi-agent collaboration.

[0141] In this embodiment, the agent training framework calls at least one reward calculation model, synchronously inputting the target return data and the corresponding real return data into the reward calculation model. The model performs multi-dimensional quantitative analysis on the two sets of data from dimensions such as data integrity, logical consistency, and business result fit, and outputs a numerical data gap representing the degree of difference between the two. According to the preset reward calculation rules, based on the size of the data gap and the differentiation rules corresponding to the business type, the agent training framework completes the conversion of data gap into task reward value according to the core logic that the smaller the data gap, the higher the reward value, and generates the task reward value corresponding to each target return data. Finally, the framework uses a preset group relative strategy optimization algorithm, with the task reward value as the core optimization basis, and combines the functional division of the agent to be trained within the group (such as data query, visualization generation, etc.) and the real-time training progress to make targeted and differentiated adjustments to the model weights of the agent to be trained associated with the target return data. During the adjustment process, the reward value performance of other agents in the group is taken into account to balance the needs of single agent performance improvement and multi-agent collaborative optimization. The model weights are adjusted by using a group-relative strategy optimization algorithm, which balances the performance improvement of single agents with the synergy of multiple agents. This avoids the problem of intra-group synergy failure caused by over-optimization of single agents and is suitable for the business characteristics of multi-agent collaboration to complete complex tasks.

[0142] Specifically, in the actual training process, in order to save GPU memory and improve fine-tuning speed, the KL divergence calculation between the policy model and the reference model is cancelled, and the objective function for updating the model weights is as follows:

[0143]

[0144] Where θ: the parameters of the current strategy model to be optimized.

[0145] θ old : The parameters of the old strategy model from the previous iteration, used for importance sampling.

[0146] π θ : The current strategy, i.e., the language model, is defined by the parameter θ.

[0147] π θold : Old strategy, used to calculate the strategy ratio.

[0148] π ref: Reference policy, usually a pre-trained model or an earlier policy, used for KL divergence regularization.

[0149] q: Input query (prompt), which is the input to the LLM.

[0150] o i,t : The output sequence of the t-th token in the i-th sample group.

[0151] o i : The complete output sequence of the i-th sample group, with length ∣o i ∣.

[0152] P(Q): Query distribution, which is the distribution of queries in the training data.

[0153] G: Group size sampled for each query, that is, the number of candidate outputs generated for the same query.

[0154] ∣o i ∣: The sequence length of the i-th output sequence o i '.

[0155] : Expectation symbol, indicating averaging over the output groups sampled from the query distribution P(Q) and the old policy πθold.

[0156] : Policy ratio, measuring the relative change in the predicted probability of the next token oi,t for the current policy compared to the old policy given the context oi,<t.

[0157] : Advantage function estimate, representing the advantage of the token oi,t relative to the average performance within the group. It is obtained by subtracting the mean reward within the group from the reward ri of the sequence where the token is located, and then dividing by the standard deviation of the rewards within the group.

[0158] ε: Clipping coefficient, a small hyperparameter (such as 0.2), used to limit the range of the policy ratio.

[0159] β: KL divergence penalty coefficient, controlling the strength of the KL divergence regularization term and balancing reward maximization and policy stability.

[0160] D KL (π θ ∥π refKL divergence (KL divergence) measures the current policy π. θ With reference strategy π ref The difference between them is used as a regularization term to prevent the model from deviating too far from the pre-training distribution.

[0161] It should be noted that a hierarchical strategy can be used for updating the model weights in this invention, including:

[0162] 1. High-level credit allocation: It first distributes the final reward of the entire task to each LLM call in the trajectory. In the simplest implementation, it directly makes the reward for each call equal to the final reward.

[0163] 2. Low-level policy update: With a reward at each step, each LLM call (input, output, reward) becomes an independent single-call RL problem. At this point, any existing, mature single-call RL algorithm such as GRPO can be directly applied to update the model parameters.

[0164] Within the framework of Reinforcement Learning (RL), a truly good action will systematically and with a higher probability appear in the final successful task. After thousands of training iterations, it will be associated with positive rewards far more often than negative rewards, and vice versa. Although the signal-to-noise ratio of a single assignment is low, given a sufficient number of training samples, the correct signal will eventually emerge.

[0165] Step 105: Adjust the difficulty of the training task set according to the reward value of each task to obtain a new training task set;

[0166] In this embodiment, while adjusting the model weights of the agent to be trained, the model training framework can also divide all training tasks into difficulty levels and group them according to the numerical range matched by the reward values ​​of each task. For task samples of the first preset difficulty, similar samples are automatically generated and deduplicated. The expanded samples are assigned to the expanded task group. Then, combined with the noise sample filtering rules, invalid training tasks are eliminated according to the number of rewards and the average reward value of each task. Subsequently, according to the preset difficulty task ratio, the tasks of each difficulty group are screened and reorganized. At the same time, the reorganized task set is randomly shuffled to obtain a new training task set.

[0167] In one example of the present invention, step 105 may include the following sub-steps:

[0168] The task difficulty of the training task corresponding to each task reward value is determined according to the numerical range in which each task reward value is located.

[0169] The training tasks are clustered according to their difficulty to obtain multiple task groups;

[0170] The training tasks within each task group are reorganized according to the preset reorganization logic to obtain multiple update task groups;

[0171] Select tasks to be expanded from the update task group whose task difficulty meets the first preset difficulty, and expand the tasks to be expanded according to the preset expansion rules to obtain an expansion task group containing multiple expansion tasks.

[0172] Select the first task from the update task group whose task difficulty meets the second preset difficulty, and construct the first task subset by selecting the first task in the first proportion.

[0173] Select a second task from the expanded task group at a second ratio, and combine it with an updated task group whose difficulty meets the first preset difficulty to construct a second task subset; the ratio between the first task subset and the second task subset is a preset construction ratio;

[0174] A new training task set is generated by using the first task subset and the second task subset.

[0175] A numerical range refers to a continuous range of values ​​used to divide task reward values. Different ranges correspond to different task difficulties in the training task. Task difficulty is the complexity assessed based on the task reward value of the corresponding training task, and may include, but is not limited to, easy, medium, and hard.

[0176] The tasks to be expanded refer to the core training tasks that are selected from the first preset difficulty update task group and have the value of sample expansion.

[0177] In this embodiment, the agent training framework maps the numerical range of each task reward value to the corresponding task difficulty, and merges the training tasks according to the task difficulty mapped to each task reward value to obtain multiple task groups with similar difficulty features. For each task group, the agent training framework performs pseudo-random processing such as task rearrangement, noise filtering, and order optimization according to a preset recombination logic to generate multiple updated task groups adapted to the course learning pace.

[0178] Training tasks from the update task group with task difficulty matching a first preset difficulty (such as medium and / or hard) are selected as tasks to be expanded. These tasks are then expanded according to preset expansion rules (such as generating similar ChatBI business logic, expanding data dimensions, or generating large models followed by deduplication), forming an expansion task group containing multiple expanded tasks. During this task expansion process, the number of tasks expanded each time and the total number of tasks can be limited.

[0179] The framework selects the first task from the updated task group with a task difficulty matching the second preset difficulty (e.g., easy) according to the first ratio and integrates them into the first task subset. It then selects the second task from the expanded task group according to the second ratio and integrates them with all the updated task groups with the first preset difficulty to form the second task subset, ensuring that the ratio of the first and second task subsets conforms to the preset construction ratio, such as 3:7. Finally, the framework mixes and recombines the two task subsets according to the preset construction ratio to generate a new training task set that adapts to the learning pace of the agent course.

[0180] For example, the task difficulty of each training task is determined based on its reward value: [0, 0.3) is easy, [0.3, 0.7) is medium, and [0.7, 1.0) is hard. Tasks are then divided into three groups: Easy, Medium, and Hard. After reorganizing the tasks within each group, an updated task group is obtained. Tasks to be expanded are selected from the first preset difficulty group (Medium / Hard), generated using a large model, and then deduplicated to expand the tasks, resulting in an expanded task group Uknow containing multiple expanded tasks. After obtaining Uknow, the first task subset is constructed by selecting a first proportion x% of the tasks from the updated task group whose difficulty meets the second preset difficulty (Easy). Simultaneously, a second task at a second proportion (y%) is selected from the expanded task group, and combined with the updated task group whose difficulty meets the first preset difficulty (Medium / Hard), to construct a second task subset; the ratio between the first and second task subsets is a preset construction ratio; this preset construction ratio can be Easy:Medium+Hard+Uknow=3:7. A new training task set is generated using the first and second task subsets.

[0181] Furthermore, step 105 may also include the following sub-steps:

[0182] If the average reward value of any training task is lower than the preset reward threshold for several consecutive rounds, the training task is removed.

[0183] In this embodiment, the agent training framework records the task reward value obtained by each training task after it is executed in multiple consecutive allocation rounds in real time. For each training task, the arithmetic mean of its reward value in these multiple rounds is calculated. The calculated mean is compared with a preset reward threshold one by one. If it is detected that the mean reward value of any training task is continuously lower than the reward threshold in the set multiple allocation rounds, it is determined that the training task has no effective training value (such as having business logic defects, difficulty far exceeding the current ability level of the agent, or being an invalid noise task). Then, the task removal mechanism is triggered to permanently remove the training task from the current training task set.

[0184] For example, if a hard task is trained n times but the average reward is still low, it can be inferred that the samples are either extremely difficult or simply noise and need to be removed. Hard and medium samples will decrease, while easy samples will increase, allowing the agent to focus primarily on learning from medium and hard samples, thus continuously improving model performance.

[0185] Step 106: Jump to execute the step of assigning each training task in the training task set to the agent to be trained one by one, until the number of assignment rounds reaches the preset assignment threshold or the number of tasks in the training task set is less than the preset batch threshold, and the agent to be trained at the current moment is determined as the target agent.

[0186] The preset assignment threshold refers to the maximum number of task assignment rounds set for the reinforcement learning training of the agents to be trained. It is one of the criteria for determining the number of rounds to terminate the training process. The assignment thresholds for each agent to be trained can be the same or different.

[0187] The batch threshold refers to the minimum number of tasks set for the training task set. When the number of tasks is lower than this value, it cannot support a batch of reinforcement learning training, and it is one of the criteria for terminating the training process.

[0188] The target agent refers to the agent to be trained that has completed multiple rounds of reinforcement learning training, optimized model weights, and whose business execution capabilities meet actual business needs and can be adapted to the online operating environment.

[0189] In this embodiment, after the agent training framework completes the reorganization and update of the training task set, steps 101-105 are executed iteratively. The new training task set is then matched and allocated to the corresponding agents according to their functional divisions. Subsequent tasks such as task request forwarding, target return data generation, task reward value calculation, model weight adjustment, and task set difficulty adjustment are carried out sequentially. After each round of task allocation, the current allocation round number is counted in real time, and the number of remaining tasks in the training task set is checked. If the allocation round number reaches the preset allocation threshold, or the number of tasks in the training task set is less than the batch threshold, the reinforcement learning training process is terminated immediately, and the agent whose model performance meets the standard and can adapt to the business environment is identified as the target agent.

[0190] Furthermore, the task reward values ​​for each training task can be obtained by requesting intermediate service components through the agent training framework. Additionally, the allocation, reorganization, and updating of training tasks can be implemented using a separate task assignment module, TaskAssign. This module can be set up within the agent training framework or independently of it.

[0191] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0192] The following describes a multi-agent reinforcement learning method apparatus for supporting course learning provided by an embodiment of the present invention. The multi-agent reinforcement learning method apparatus for supporting course learning described below can be referred to in correspondence with the multi-agent reinforcement learning method for supporting course learning described above.

[0193] Please see Figure 3 The present invention also provides a multi-agent reinforcement learning method apparatus to support course learning, applied to an agent training framework, wherein the agent training framework, an intermediate service component, and multiple agents to be trained are interconnected and communicate with each other. The apparatus includes:

[0194] The task request generation module 301 is used to allocate each training task in the training task set to each agent to be trained one by one when a training task set is received, and generate corresponding task requests through the agents to be trained.

[0195] The request forwarding module 302 is used to respond to any task request through the intermediate service component and forward the task request to the corresponding large language model according to the address variable in the task request;

[0196] The data return module 303 is used to receive the execution data returned by the large language model through the intermediate service component and convert it into a framework-readable format together with the task request to generate the target return data;

[0197] The model weight adjustment module 304 is used to calculate the task reward value corresponding to the return data of each target and adjust the model weight of the agent to be trained associated with the return data of the target.

[0198] The task set update module 305 is used to adjust the difficulty of the training task set according to the reward value of each task, so as to obtain a new training task set.

[0199] The agent loop training module 306 is used to jump to execute the step of assigning each training task in the training task set to the agent to be trained one by one until the number of allocation rounds reaches the preset allocation threshold or the number of tasks in the training task set is less than the preset batch threshold, and the agent to be trained at the current moment is determined as the target agent.

[0200] Optionally, the apparatus further includes a collection request forwarding module, specifically used for:

[0201] Send the set of model addresses corresponding to each training task to the intermediate service component;

[0202] When the intermediate service component receives any task request, it matches the dynamic interface address corresponding to the address variable from the model address set.

[0203] The intermediate service component updates the current component's output interface address to a dynamic interface address;

[0204] Forward the task request to the large language model corresponding to the output interface address of the current component.

[0205] Optionally, it also includes a large model proxy component, which is associated with multiple large language models; the large model proxy component is communicatively connected to the agent training framework and the intermediate service component respectively; the device also includes a large model proxy forwarding module, used for:

[0206] The large model proxy component responds to task requests sent by the intermediate service component and sends the task requests to the corresponding large language model according to the dynamic interface address or address variable.

[0207] When the large model proxy component receives execution data from the large language model, it returns the execution data to the intermediate service component.

[0208] Optionally, the task request generation module 301 is specifically used for:

[0209] When a training task set is received, the training tasks in the training task set are reorganized according to the preset reorganization logic to obtain a new training task set.

[0210] Each training task in the training task set is assigned to each agent to be trained.

[0211] The training task is structured and loaded into a structured template by the agent to be trained, generating the task request corresponding to the training task.

[0212] Optionally, the request forwarding module 302 is specifically used for:

[0213] The intermediate service component responds to any task request and extracts the payload within the task request.

[0214] The address variable matching the variable name is retrieved from the payload through the intermediate service component;

[0215] The intermediate service component updates the current component's output interface address to the address variable.

[0216] The intermediate service component forwards the task request to the large language model corresponding to the output interface address of the current component.

[0217] Optionally, the model weight adjustment module 304 is specifically used for:

[0218] Call at least one reward calculation model to calculate the data gap between the target return data and the actual return data;

[0219] Calculate the task reward value for each target's returned data based on the data gap and the preset reward calculation rules;

[0220] A pre-defined group-relative strategy optimization algorithm is used to adjust the model weights of the agent to be trained in relation to the target return data according to the reward value of each task.

[0221] Optionally, the task set update module 305 is specifically used for:

[0222] The task difficulty of the training task corresponding to each task reward value is determined according to the numerical range in which each task reward value is located.

[0223] The training tasks are clustered according to their difficulty to obtain multiple task groups;

[0224] The training tasks within each task group are reorganized according to the preset reorganization logic to obtain multiple update task groups;

[0225] Select tasks to be expanded from the update task group whose task difficulty meets the first preset difficulty, and expand the tasks to be expanded according to the preset expansion rules to obtain an expansion task group containing multiple expansion tasks.

[0226] Select the first task from the update task group whose task difficulty meets the second preset difficulty, and construct the first task subset by selecting the first task in the first proportion.

[0227] Select a second task from the expanded task group at a second ratio, and combine it with an updated task group whose difficulty meets the first preset difficulty to construct a second task subset; the ratio between the first task subset and the second task subset is a preset construction ratio;

[0228] A new training task set is generated by using the first task subset and the second task subset.

[0229] Optionally, the task set update module 305 is also used for:

[0230] If the average reward value of any training task is lower than the preset reward threshold for several consecutive rounds, the training task is removed.

[0231] Optionally, the device also includes a re-invocation judgment module, specifically used for:

[0232] The execution data is returned to the agent to be trained via an intermediate service component;

[0233] The agent to be trained performs logical analysis on the execution data, and determines whether there is a need for re-calling based on the analysis results.

[0234] If so, a new task request is generated according to the analysis results and sent to the intermediate service component. The process then jumps to the steps of responding to any task request through the intermediate service component and forwarding the task request to the corresponding large language model according to the address variable in the task request.

[0235] If not, then all execution data is integrated and sent to the intermediate service component.

[0236] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device and module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0237] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0238] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0239] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0240] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-agent reinforcement learning method for supporting course learning, characterized in that, An application to an agent training framework, wherein the agent training framework communicates and connects with intermediate service components and multiple agents to be trained, the method comprising: When a training task set is received, each training task in the training task set is assigned to each of the agents to be trained, and the agents to be trained generate corresponding task requests. The intermediate service component responds to any of the task requests and forwards the task request to the corresponding large language model according to the address variable in the task request. The intermediate service component receives the execution data returned by the large language model and converts it into a framework-readable format along with the task request to generate the target return data. Calculate the task reward value corresponding to each of the target return data, and adjust the model weights of the agent to be trained associated with the target return data; The difficulty of the training task set is adjusted according to the reward value of each task to obtain a new training task set. Jump to execute the step of assigning each training task in the training task set to the agent to be trained one by one, until the number of allocation rounds reaches a preset allocation threshold or the number of tasks in the training task set is less than a preset batch threshold, and the agent to be trained at the current moment is determined as the target agent.

2. The multi-agent reinforcement learning method for supporting course learning according to claim 1, characterized in that, Also includes: Send the set of model addresses corresponding to each training task to the intermediate service component; When the intermediate service component receives any of the task requests, it matches the dynamic interface address corresponding to the address variable from the model address set; The intermediate service component updates the current component's output interface address to a dynamic interface address. The task request is forwarded to the large language model corresponding to the output interface address of the current component.

3. The multi-agent reinforcement learning method for supporting course learning according to claim 2, characterized in that, It also includes a large model proxy component, which is associated with multiple large language models; the large model proxy component is communicatively connected to the agent training framework and the intermediate service component respectively; the method further includes: The large model proxy component responds to the task request sent by the intermediate service component and sends the task request to the corresponding large language model according to the dynamic interface address or the address variable. When the large model proxy component receives execution data from the large language model, it returns the execution data to the intermediate service component.

4. The multi-agent reinforcement learning method for supporting course learning according to claim 1, characterized in that, The step of assigning each training task in the training task set to each of the agents to be trained when a training task set is received, and generating a corresponding task request through the agent to be trained, includes: When a training task set is received, the training tasks in the training task set are reorganized according to a preset reorganization logic to obtain a new training task set. Each training task in the training task set is assigned to each of the intelligent agents to be trained; The training task is structured and loaded into a structured template by the agent to be trained, thereby generating a task request corresponding to the training task.

5. The multi-agent reinforcement learning method for supporting course learning according to claim 1, characterized in that, The step of responding to any of the task requests through the intermediate service component and forwarding the task request to the corresponding large language model according to the address variable in the task request includes: The intermediate service component responds to any of the task requests and extracts the payload within the task request. The intermediate service component retrieves the address variable matching the variable name from within the payload; The intermediate service component updates the current component's output interface address to an address variable. The intermediate service component forwards the task request to the large language model corresponding to the output interface address of the current component.

6. The multi-agent reinforcement learning method for supporting course learning according to claim 1, characterized in that, The steps of calculating the task reward value corresponding to each of the target return data and adjusting the model weights of the agent to be trained associated with the target return data include: At least one reward calculation model is invoked to calculate the data gap between the target return data and the actual return data; Based on the data gap and the preset reward calculation rules, calculate the task reward value for each of the target returned data; A preset group-relative strategy optimization algorithm is used to adjust the model weights of the agent to be trained associated with the target return data according to the reward values ​​of each task.

7. The multi-agent reinforcement learning method for supporting course learning according to claim 1, characterized in that, The step of adjusting the difficulty of the training task set according to the reward value of each task to obtain a new training task set includes: The task difficulty of the training task corresponding to each task reward value is determined according to the numerical range in which each task reward value is located. The training tasks are clustered according to their difficulty to obtain multiple task groups; The training tasks within each task group are reorganized according to a preset reorganization logic to obtain multiple updated task groups. Select tasks to be expanded from the updated task group whose task difficulty meets the first preset difficulty, and expand the tasks to be expanded according to the preset expansion rules to obtain an expanded task group containing multiple expanded tasks. Select a first task from the updated task group whose task difficulty meets the second preset difficulty, and construct a first task subset; A second task subset is constructed by selecting a second proportion of the expanded task group and combining it with all the updated task groups whose difficulty meets the first preset difficulty; the ratio between the first task subset and the second task subset is a preset construction ratio. A new training task set is generated by using the first task subset and the second task subset.

8. The multi-agent reinforcement learning method for supporting course learning according to claim 1 or 7, characterized in that, Also includes: If the average reward value of any of the training tasks is lower than a preset reward threshold for multiple consecutive allocation rounds, then the training task is removed.

9. The multi-agent reinforcement learning method for supporting course learning according to claim 1, characterized in that, After performing the step of receiving the execution data returned by the large language model through the intermediate service component, the method further includes: The execution data is returned to the agent to be trained via the intermediate service component. The training agent performs logical analysis on the execution data and determines whether there is a need for re-calling based on the analysis results. If so, a new task request is generated according to the analysis results and sent to the intermediate service component. Then, the process jumps to the step of responding to any of the task requests through the intermediate service component and forwarding the task request to the corresponding large language model according to the address variable in the task request. If not, then all the execution data is integrated and sent to the intermediate service component.

10. A multi-agent reinforcement learning method device for supporting course learning, characterized in that, An apparatus for use in an agent training framework, wherein the agent training framework communicates and connects with an intermediate service component and multiple agents to be trained, the apparatus comprising: The task request generation module is used to, when a training task set is received, allocate each training task in the training task set to each of the intelligent agents to be trained, and generate corresponding task requests through the intelligent agents to be trained. The request forwarding module is used to respond to any of the task requests through the intermediate service component and forward the task request to the corresponding large language model according to the address variable in the task request; The data return module is used to receive the execution data returned by the large language model through the intermediate service component and convert it into a framework-readable format together with the task request to generate the target return data; The model weight adjustment module is used to calculate the task reward value corresponding to each of the target return data and adjust the model weights of the agent to be trained associated with the target return data. The task set update module is used to adjust the difficulty of the training task set according to the reward value of each task to obtain a new training task set; The agent loop training module is used to jump to execute the step of assigning each training task in the training task set to the agent to be trained one by one until the number of allocation rounds reaches a preset allocation threshold or the number of tasks in the training task set is less than a preset batch threshold, and then determine the agent to be trained at the current moment as the target agent.

Images