Search method, reinforcement learning method and device based on multi-agent system

Abstract

The one or more embodiments of the specification provide a multi-agent system-based search method, a reinforcement learning method and device. Through the method, user query information is first acquired, and a shared memory pool is initialized; then a planning agent is called to generate a search word based on the user query information; thereafter, a search engine is called to search original retrieval data corresponding to the search word; a filtering agent is called to extract a relevant text segment from the original retrieval data and store the relevant text segment into the shared memory pool; finally, an inference agent is called to generate response information corresponding to the user query information based on the data stored in the shared memory pool.

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence technology, and in particular to a search method, reinforcement learning method and apparatus based on a multi-agent system. Background Technology

[0002] In scenarios where complex reasoning tasks are handled through deep search and interaction with the external environment, when faced with multi-hop complex search tasks, artificial intelligence models (hereinafter referred to as AI models) are forced to handle multiple tasks in the same context.

[0003] This multitasking burden can lead to severe cognitive overload. Specifically, as the number of retrieval steps increases, the accumulating noisy text distracts the model, causing it to lose its reasoning chain in long contexts. Ultimately, even if the AI ​​model obtains the correct information, it may experience factual illusions or forget key information. Summary of the Invention

[0004] To address the cognitive overload problem of AI models, this specification provides one or more embodiments of a search method, reinforcement learning method, and apparatus based on a multi-agent system.

[0005] Firstly, one or more embodiments of this specification provide a search method based on a multi-agent system. The multi-agent system includes a planning agent, a filtering agent, and an inference agent. The method includes: acquiring user query information and initializing a shared memory pool; invoking the planning agent to generate search terms based on the user query information; invoking a search engine to search for original retrieval data corresponding to the search terms; invoking the filtering agent to extract relevant text fragments from the original retrieval data and storing the relevant text fragments in the shared memory pool; and invoking the inference agent to generate response information corresponding to the user query information based on the data stored in the shared memory pool.

[0006] In one possible implementation, the planning agent is invoked to generate search terms based on user query information, including: in the first round of invoking the planning agent, generating a first search term based on user query information; in the Nth round of invoking the planning agent, generating a second search term based on user query information, historical search records, and data stored in the shared memory pool, where N is an integer greater than 1, and the historical search records include search terms generated by the planning agent in each round prior to the Nth round for user query information.

[0007] In one possible implementation, the method further includes: stopping the generation of search terms when it is determined that the search stopping conditions are met based on user query information, historical search records, and data stored in a shared memory pool.

[0008] In one possible implementation, the filtering agent is invoked to extract relevant text fragments from the original retrieval data, including: guiding the filtering agent to extract relevant text fragments related to the search terms based on preset prompts.

[0009] In one possible implementation, the method further includes: compressing the data in the shared memory pool when the amount of data in the shared memory pool exceeds a threshold.

[0010] In one possible implementation, when the amount of data in the shared memory pool exceeds a threshold, the data in the shared memory pool is compressed, including: compressing the data in the shared memory pool into atomic facts using a pre-trained semantic compression model, or compressing the data in the shared memory pool by extracting a summary.

[0011] In one possible implementation, the inference agent is invoked to generate response information corresponding to the user query based on the data stored in the shared memory pool. This includes: when the planning agent stops generating search terms or the shared memory pool reaches its maximum storage capacity, the inference agent is invoked to generate response information corresponding to the user query based on the data stored in the shared memory pool.

[0012] In one possible implementation, at least two of the planning agent, filtering agent, and reasoning agent share the same artificial intelligence model or adapter; or, the planning agent, filtering agent, and reasoning agent are each configured with an independent artificial intelligence model or adapter.

[0013] Secondly, one or more embodiments of this specification also provide a reinforcement learning method for a multi-agent system. The method includes: acquiring training data, which includes pairs of query information and standard response information; jointly training the multi-agent system based on the training data, the multi-agent system comprising multiple cooperating agents, with different agents performing different tasks; wherein, during the joint training process, the following steps are included: sampling multiple different trajectories for the same query information; for each trajectory, calculating the mixed reward corresponding to each agent in the multi-agent system, the mixed reward including format reward, team result reward, and role-specific reward, the role-specific reward being used to characterize the local contribution of the corresponding agent; using a group policy optimization algorithm, calculating the advantage value of each agent in each trajectory based on the mixed reward; and updating the model parameters corresponding to the multi-agent system based on the advantage value.

[0014] In one possible implementation, the multi-agent system includes a planning agent, a filtering agent, and an inference agent; wherein, the planning agent is used to generate search terms based on the input query information, so that the search engine can search for the original retrieval data corresponding to the search terms; the filtering agent is used to extract relevant text fragments from the original retrieval data and store the relevant text fragments in a shared memory pool; and the inference agent is used to generate corresponding predicted response information based on the data stored in the shared memory pool.

[0015] In one possible implementation, calculating the role-specific reward for each agent in a multi-agent system includes: using a large evaluation model to calculate the role-specific reward for each agent based on the input and output information of each agent in the multi-agent system.

[0016] In one possible implementation, a group policy optimization algorithm is used to calculate the advantage value of each agent in each trajectory based on the mixed reward. This includes: grouping the mixed rewards corresponding to agents of the same role type in multiple trajectories sampled for the same query information into multiple role groups; calculating the mean and standard deviation of the mixed reward within each role group; and calculating the advantage value of the corresponding agent in each trajectory based on the mixed reward and the mean and standard deviation of the group to which the mixed reward belongs.

[0017] In one possible implementation, the agent's advantage value in each trajectory is calculated based on the mixed reward and the mean and standard deviation of the group to which the mixed reward belongs, including: The advantage value of the agent in each trajectory is calculated using the following formula: =

[0018] in, Let represent the advantage value of agent u in the i-th trajectory. Let represent the mixed reward of agent u in the i-th trajectory. and Let these represent the mean and variance of agent u in its role group, respectively. It is a constant.

[0019] In one possible implementation, the agents in a multi-agent system share the same artificial intelligence model or adapter; wherein, updating the model parameters corresponding to the multi-agent system based on the advantage value includes: merging the advantage values ​​corresponding to each agent in the multi-agent system, and jointly updating the parameters of the shared artificial intelligence model or adapter based on the merged advantage value.

[0020] In one possible implementation, each agent in the multi-agent system is configured with an independent artificial intelligence model or adapter; wherein, updating the model parameters of the multi-agent system based on the advantage value includes: updating the parameters of the corresponding configured artificial intelligence model or adapter based on the advantage value of each agent.

[0021] Thirdly, one or more embodiments of this specification also provide a multi-agent system, the system comprising: a planning agent for generating search terms based on user query information, for a search engine to search for original retrieval data corresponding to the search terms; a filtering agent for extracting relevant text fragments from the original retrieval data and storing the relevant text fragments in a shared memory pool; and a reasoning agent for generating response information corresponding to the user query information based on the data stored in the shared memory pool.

[0022] In one possible implementation, a planning agent is used to generate a first search term based on the user query information in the first round of calling the planning agent; and to generate a second search term based on the user query information, historical search records, and data stored in the shared memory pool in the Nth round of calling the planning agent, where N is an integer greater than 1, and the historical search records include search terms generated by the planning agent in each round before the Nth round for the user query information.

[0023] In one possible implementation, the planning agent is also used to stop generating search terms when it is determined that the search stopping conditions are met, based on user query information, historical search records, and data stored in a shared memory pool.

[0024] In one possible implementation, a filtering agent is used to guide the filtering agent to extract relevant text fragments related to the search term based on preset prompts.

[0025] In one possible implementation, the reasoning agent is used to generate response information corresponding to the user query information based on the data stored in the shared memory pool, when the planning agent stops generating search terms or the shared memory pool reaches its maximum storage capacity.

[0026] In one possible implementation, at least two of the planning agent, filtering agent, and reasoning agent share the same artificial intelligence model or adapter; or, the planning agent, filtering agent, and reasoning agent are each configured with an independent artificial intelligence model or adapter.

[0027] Fourthly, one or more embodiments of this specification also provide a search device based on a multi-agent system. The multi-agent system includes a planning agent, a filtering agent, and an inference agent. The device includes: an acquisition module for acquiring user query information and initializing a shared memory pool; a planning module for calling the planning agent to generate search terms based on the user query information; a search module for calling a search engine to search for the original retrieval data corresponding to the search terms; a filtering module for calling the filtering agent to extract relevant text fragments from the original retrieval data and store the relevant text fragments in the shared memory pool; and an inference module for calling the inference agent to generate response information corresponding to the user query information based on the data stored in the shared memory pool.

[0028] In one possible implementation, the planning module is used to generate a first search term based on the user query information when the planning agent is called in the first round; and to generate a second search term based on the user query information, historical search records, and data stored in the shared memory pool when the planning agent is called in the Nth round, where N is an integer greater than 1, and the historical search records include search terms generated by the planning agent in each round before the Nth round for the user query information.

[0029] In one possible implementation, the planning module is further configured to stop generating search terms when it determines that the search stopping conditions are met, based on user query information, historical search records, and data stored in a shared memory pool.

[0030] In one possible implementation, a filtering module is used to guide the filtering agent to extract relevant text fragments related to the search term based on preset prompts.

[0031] One possible implementation also includes a compression module for compressing data in the shared memory pool when the amount of data in the shared memory pool exceeds a threshold.

[0032] In one possible implementation, a compression module is used to compress the data in the shared memory pool into atomic facts using a pre-trained semantic compression model when the amount of data in the shared memory pool exceeds a threshold, or to compress the data by extracting a summary from the data in the shared memory pool.

[0033] In one possible implementation, the inference module is used to invoke the inference agent to generate response information corresponding to the user query information based on the data stored in the shared memory pool when the planning agent stops generating search terms or the shared memory pool reaches its maximum storage capacity.

[0034] In one possible implementation, at least two of the planning agent, filtering agent, and reasoning agent share the same artificial intelligence model or adapter; or, the planning agent, filtering agent, and reasoning agent are each configured with an independent artificial intelligence model or adapter.

[0035] Fifthly, one or more embodiments of this specification also provide a reinforcement learning apparatus for a multi-agent system, comprising: an acquisition module for acquiring training data, the training data including pairs of query information and standard response information; and a training module for jointly training the multi-agent system based on the training data, the multi-agent system including multiple cooperating agents, and different agents performing different tasks. The training module is specifically used for: sampling multiple different trajectories for the same query information; for each trajectory, calculating the mixed reward corresponding to each agent in the multi-agent system, which includes format reward, team result reward, and role-specific reward, with the role-specific reward used to represent the local contribution of the corresponding agent; using the group policy optimization algorithm, calculating the advantage value of each agent in each trajectory based on the mixed reward; and updating the model parameters corresponding to the multi-agent system based on the advantage value.

[0036] In one possible implementation, the multi-agent system includes a planning agent, a filtering agent, and an inference agent; wherein, the planning agent is used to generate search terms based on the input query information, so that the search engine can search for the original retrieval data corresponding to the search terms; the filtering agent is used to extract relevant text fragments from the original retrieval data and store the relevant text fragments in a shared memory pool; and the inference agent is used to generate corresponding predicted response information based on the data stored in the shared memory pool.

[0037] In one possible implementation, the training module is specifically used to calculate the role-specific reward for each agent based on the input and output information of each agent in the multi-agent system using the evaluation model.

[0038] In one possible implementation, the training module is specifically used to group the mixed rewards corresponding to agents of the same role type in multiple trajectories sampled for the same query information into multiple role groups; calculate the mean and standard deviation of the mixed rewards in each role group; and calculate the advantage value of the corresponding agent in each trajectory based on the mixed rewards and the mean and standard deviation of the group to which the mixed rewards belong.

[0039] In one possible implementation, the training module is specifically used to calculate the agent's advantage value in each trajectory using the following formula: =

[0040] in, Let represent the advantage value of agent u in the i-th trajectory. Let represent the mixed reward of agent u in the i-th trajectory. and Let these represent the mean and variance of agent u in its role group, respectively. It is a constant.

[0041] In one possible implementation, the agents in the multi-agent system share the same artificial intelligence model or adapter; the training module is specifically used to merge the advantage values ​​of each agent in the multi-agent system, and jointly update the parameters of the shared artificial intelligence model or adapter based on the merged advantage values.

[0042] In one possible implementation, each agent in the multi-agent system is configured with an independent artificial intelligence model or adapter; the training module is specifically used to update the parameters of the corresponding artificial intelligence model or adapter based on the advantage value of each agent.

[0043] In a sixth aspect, one or more embodiments of this specification also provide an electronic device, the electronic device including a memory and a processor; the memory is used to store a computer program product; the processor is used to execute the computer program product stored in the memory, and when the computer program product is executed, it implements any of the methods of the first or second aspect described above.

[0044] In a seventh aspect, one or more embodiments of this specification also provide a computer-readable storage medium storing computer program instructions that, when executed, implement any of the methods described in the first or second aspect.

[0045] Eighthly, one or more embodiments of this specification also provide a computer program product comprising one or more programs that, when run on an electronic device, cause the electronic device to implement any of the methods described in the first or second aspect.

[0046] In summary, this specification provides a search method, reinforcement learning method, and apparatus based on a multi-agent system through one or more embodiments. In this method, planning, filtering, and reasoning tasks are assigned to independent agents through hard decoupling of roles. The planning agent and the reasoning agent only observe high-value information in the shared memory pool after filtering, physically shielding them from original retrieval noise and the history of search failures, allowing each to perform its specialized task within a clean information field. This approach effectively avoids attentional distraction and loss of reasoning chains caused by long contexts, alleviates cognitive overload, and enables the model to maintain accurate reasoning ability even in the face of significant noise generated by multiple rounds of retrieval, thereby significantly reducing the probability of factual illusions. Attached Figure Description

[0047] To more clearly illustrate the technical solutions of one or more embodiments of this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of one or more embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 An architecture diagram of a multi-agent system provided for one or more embodiments of this specification; Figure 2 A flowchart illustrating a search method based on a multi-agent system provided for one or more embodiments of this specification; Figure 3 A flowchart illustrating a reinforcement learning method for a multi-agent system provided in one or more embodiments of this specification; Figure 4 A flowchart illustrating one implementation of step S204 provided for one or more embodiments of this specification; Figure 5 A set of experimental data provided for one or more embodiments of this specification; Figure 6 A structural block diagram of a search device based on a multi-agent system provided for one or more embodiments of this specification; Figure 7 A structural block diagram of a reinforcement learning device for a multi-agent system provided for one or more embodiments of this specification; Figure 8 This is a structural block diagram of an electronic device provided for one or more embodiments of this specification. Detailed Implementation

[0049] The present specification describes one or more embodiments in further detail below with reference to the accompanying drawings and examples. Through these descriptions, the features and advantages of one or more embodiments of the present specification will become clearer and more apparent.

[0050] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments. Although various aspects of embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless specifically indicated otherwise.

[0051] Furthermore, the technical features described below in one or more embodiments of this specification may be combined with each other as long as they do not conflict with each other.

[0052] It should be noted that the user information and data involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties.

[0053] The data acquisition and collection actions involved in one or more embodiments of this specification are all performed after authorization by the user, the object, or after full authorization by all parties.

[0054] The user information processed in one or more embodiments of this specification may vary depending on the specific product / service scenario. The specific scenario in which the user uses the product / service shall prevail, and may involve user account information, device information, driving information, vehicle information, or other related information. This application will treat user's personal information and its processing with the utmost diligence.

[0055] This specification and one or more embodiments place great emphasis on the security of user information and have adopted reasonable and feasible security protection measures that comply with industry standards to protect user information and prevent unauthorized access, public disclosure, use, modification, damage or loss of user information.

[0056] To facilitate understanding, the application scenarios of the technical solutions provided in one or more embodiments of this specification will be described below.

[0057] Artificial intelligence models (hereinafter referred to as AI models), such as large language models (LLM), refer to deep learning models with a large number of parameters that can understand and generate natural language.

[0058] Deep search refers to the process by which AI models autonomously plan retrieval strategies, extract information from massive external corpora, and perform multi-step reasoning to answer complex queries.

[0059] Currently, in scenarios where deep search interacts with the external environment to handle complex reasoning tasks, AI models are forced to process multiple tasks within the same context window when faced with multi-hop complex search tasks.

[0060] This multitasking burden can lead to severe cognitive overload. Specifically, as the number of retrieval steps increases and the recall rate improves, the ever-accumulating noisy text distracts the model, causing it to lose its reasoning chain in long contexts. Ultimately, this results in the AI ​​model experiencing factual illusions or forgetting key information even when it obtains the correct information.

[0061] To address the cognitive overload problem of the aforementioned AI models, this specification provides one or more embodiments of a search method based on a multi-agent system. This method hard decouples the planning, filtering, and reasoning roles to construct a multi-agent system comprising a planning agent, a filtering agent, and a reasoning agent.

[0062] The method includes: acquiring user query information and initializing a shared memory pool; invoking a planning agent to generate search terms based on the user query information; invoking a search engine to search for raw retrieval data corresponding to the search terms; invoking a filtering agent to extract relevant text fragments from the raw retrieval data and storing the relevant text fragments in the shared memory pool; and invoking an inference agent to generate response information corresponding to the user query information based on the data stored in the shared memory pool.

[0063] In this way, unlike traditional approaches that mix multiple tasks in the same context, this method hard decouples roles, assigning planning, filtering, and reasoning tasks to independent agents. The planning agent and the reasoning agent only observe high-value information in the shared memory pool after filtering, physically shielding them from original retrieval noise and the history of failed searches, allowing each to perform its specialized task within a clean information field. This effectively avoids attentional distractions and loss of reasoning chains caused by long contexts, alleviates cognitive overload, and enables the model to maintain accurate reasoning ability even in the face of significant noise generated by multiple rounds of retrieval, thereby significantly reducing the probability of factual illusions.

[0064] Figure 1 This is an architecture diagram of a multi-agent system provided for one or more embodiments of this specification.

[0065] like Figure 1As shown, one or more embodiments of this specification provide a multi-agent system that decomposes complex question-answering tasks into three isolated agent types: a planning agent, a filtering agent, and a reasoning agent. Each agent relies on the capabilities of the underlying AI model to implement its corresponding function, and each agent maintains its own independent context space. Through this context-isolated architecture, irrelevant and redundant information is eliminated from the input context of a single agent, making the task execution objective more focused and effectively improving the system's stability in handling long-term, multi-hop complex problems.

[0066] Specifically, the raw retrieval data output by the search engine is only accessible to the filtering agent; the planning and reasoning agents cannot access it. The planning agent can only observe user queries, historical search information, and facts extracted from the shared memory pool, but cannot access the noisy text discarded by the filtering agent. The reasoning agent can only observe user queries and the final valid data in the shared memory pool, but cannot access historical search behavior, retrieval failure records, etc. The filtering agent only stores the extracted valid facts in the shared memory pool. Through this multi-layered observation and isolation design, the input context of the planning, reasoning, and inference agents remains concise and effective, blocking the inflow of raw retrieval noise into the planning and reasoning links, alleviating cognitive overload from long contexts, and effectively suppressing the model from generating answer illusions.

[0067] It should be noted that in one or more embodiments of this specification, the planning agent, filtering agent, and inference agent have flexible model deployment methods. For example, at least two of the three agents can share the same AI model or adapter; alternatively, the planning agent, filtering agent, and inference agent can each be configured with an independent AI model or independent adapter, and this embodiment does not limit this.

[0068] For example, the planning agent, filtering agent, and reasoning agent share the same AI model, and the functions are distinguished by observation isolation rules and role prompts. This eliminates the need to deploy multiple complete models and effectively saves computing resources. As an alternative, if independent AI models or independent adapters are configured for the three types of agents, it can be adapted to application scenarios with sufficient computing power.

[0069] The multi-agent system provided in one or more embodiments of this specification can be applied to any scenario that requires searching, such as a travel assistant's full-network search engine, a high-quality travel Q&A knowledge base, etc.

[0070] The following describes embodiments of the search method based on the above-described multi-agent system provided by one or more embodiments of this specification.

[0071] Figure 2 This is a flowchart illustrating a search method based on a multi-agent system provided for one or more embodiments of this specification.

[0072] See Figure 1 and Figure 2 The search method based on a multi-agent system provided in one or more embodiments of this specification may include the following steps: Step S102: Obtain user query information and initialize the shared memory pool.

[0073] After receiving user query information (e.g., natural language questions) from the user, the multi-agent system performs a shared memory pool initialization operation, clears the historical residual data in the shared memory pool, and establishes an empty data storage container.

[0074] The shared memory pool serves as the information exchange medium between the planning agent, filtering agent, and reasoning agent. It is used to uniformly store the valid evidence and entity facts (which can be referred to as relevant text fragments) extracted after each round of retrieval. Initialization ensures that the new round of question-answering tasks is not contaminated by information left over from the previous task, providing an independent and clean data space for subsequent rounds of planning, retrieval, and information storage.

[0075] After initialization, subsequent cross-agent information transmission between the planning agent, filtering agent, and reasoning agent all rely on this shared memory pool. Raw search noise is not directly written into the shared memory pool, but relevant text fragments filtered by the filtering agent are allowed to be stored.

[0076] For example, such as Figure 1 As shown, the user enters the query "Who lived longer: the author of '1984' or the creator of 'Sherlock Holmes'?". The multi-agent system retrieves this query information and performs the following initialization operations: (1) Clear the data left over from the previous round of question and answer in the shared memory pool, such as entities, text fragments, etc.; (2) Load the cleared shared memory pool into the system link and wait to receive the relevant text fragments extracted by the filtering agent; at this time there is no entity data in the shared memory pool, and prepare to start the multi-round planning retrieval loop.

[0077] In this way, by initializing and clearing historical task data, we can prevent facts and texts left over from the previous round of question and answer from being mixed into the current task, prevent old information from interfering with the reasoning in this round, and eliminate the illusion of answers caused by cross-conversation information confusion.

[0078] Step S104: Invoke the planning agent to generate search terms based on the user query information.

[0079] This step involves a planning agent undertaking multiple rounds of retrieval and decision-making tasks. The planning agent can perform one or more rounds of iterative operations, and the input information for each round changes with the iteration rounds.

[0080] For example, the first round of calling the planning agent generates the first round of search terms based on the user query information. At this time, there are no historical search records and the shared memory pool is empty.

[0081] In the Nth round, the planning agent (N > 1) is invoked: It integrates three types of information as input, including user query information, historical search records, and data stored in the shared memory pool, to generate a new round of search terms. The historical search records may include search terms generated by the planning agent in previous rounds based on the current user query information.

[0082] In each iteration, the planning agent integrates user queries, historical search records, and existing data in the shared memory pool to assess whether this information is sufficient to answer the user's question. If the existing information is sufficient, the search stops, no new search terms are generated, and the multi-round retrieval loop terminates. If the existing information is insufficient to answer the user's question, the next iteration continues.

[0083] It should be noted that, constrained by the multi-agent system framework, the planning agent cannot observe the unfiltered raw noisy documents returned by the search engine; it can only read the filtered and extracted data from the shared memory pool to avoid interference from the raw, noisy text.

[0084] For example, continue with Figure 1 For example, the planning agent is invoked to perform the following steps: 1. In the first round of calling the planning agent, the planning agent generates the first search term based on the user's original query, such as "author of novel 1984 birth and death year". The system initiates a search, and the filtering agent extracts relevant text fragments: "George Orwell (1903–1950)" and stores them in the shared memory pool.

[0085] 2. In the second round of invocation of the planning agent, the input information includes: the user's original query, historical search records (search terms from the previous round), and the relevant text fragment "George Orwell (1903–1950)" already existing in the shared memory pool. The planning agent identifies the missing information about another key figure and generates a second search term, such as "creator of Sherlock Holmes" or "date of birth and death." The system searches again, and the filtering agent extracts the relevant text fragment "A. Conan Doyle (1859–1930)" and writes it into the shared memory pool.

[0086] 3. Call the planning agent again. The planning agent reads the user query, two rounds of historical search records, and the complete birth and death facts of the two groups of people in the shared memory pool; it determines that the existing information is sufficient to compare the lifespans of the two people, meets the search stopping condition, no longer generates new search terms, and ends the multi-round retrieval process.

[0087] In this way, the planning agent filters out the massive amounts of raw, noisy documents returned by the search engine at the input level, ensuring that retrieval decisions are not interfered with by irrelevant web page text, effectively reducing the probability of generating invalid search terms and deviating from the question's objective. Simultaneously, the planning agent assesses the completeness of information based on credible evidence extracted from a shared memory pool, enabling it to accurately determine whether to stop the search: on the one hand, it avoids prematurely terminating the search when information is insufficient, leading to missing answer elements; on the other hand, it promptly ends the retrieval loop when information is sufficient, preventing infinite iteration and effectively controlling token overhead. Furthermore, the planning agent supports multi-round progressive retrieval logic: the first round initiates an initial search based on the user's query, and subsequent rounds can dynamically adjust the search direction by combining historical search records and relevant text fragments, specifically filling in missing facts, effectively solving complex multi-hop question-answering tasks that are difficult to handle with a single-round search.

[0088] Step S106: Call the search engine to search for the original retrieval data corresponding to the search terms.

[0089] The system receives search terms output by the planning agent, calls a search engine (external search tool) to perform a retrieval, and obtains multiple original web pages and text materials matching the search terms, which are recorded as the raw retrieval data. The raw retrieval data contains a large amount of mixed information: it includes relevant text fragments that support the answer to the question, but also irrelevant descriptions, redundant text, and noisy content. The raw retrieval data is not directly sent to the planning agent or the reasoning agent; it is only completely distributed to the filtering agent for cleaning and screening, strictly adhering to the information isolation rules within the framework.

[0090] For example, please continue to see Figure 1 Based on the search terms generated in the first round by the planning agent, the system calls the search engine to perform the retrieval and returns multiple pieces of raw search data, such as: DocA: George Orwell (1903–1950), the author of the novel 1984….

[0091] DocB: Film "1984" content is…….

[0092] DocC: Summary of the plot of the novel 1984….

[0093] All the above unfiltered web page texts are used as raw search data and are uniformly delivered to the filtering agent for processing.

[0094] Based on the search terms output by the second-round planning agent, the search engine is called again to obtain a set of raw documents containing mixed content such as Conan Doyle's biographical information and novel plot introductions, which are then sent to the filtering agent.

[0095] In this way, by obtaining real-time and up-to-date external information from the internet through search engines, the shortcomings of insufficient timeliness of the built-in training knowledge in AI models are overcome. This provides a factual basis for multi-hop complex question answering, reducing the risk of AI models relying on internal knowledge and creating factual illusions. The raw search data retrieved by the search engine is only provided to the filtering agent; the planning and reasoning agents cannot directly read the raw search content. This prevents massive amounts of noise from flooding the planning and reasoning processes, avoiding cognitive overload caused by redundant text from the original web pages at the architectural level.

[0096] Step S108: Call the filtering agent to extract relevant text fragments from the original retrieval data and store the relevant text fragments in the shared memory pool.

[0097] Constrained by the multi-agent system framework, the original retrieval data is only open to the filtering agent, and the extracted relevant text fragments are written into the shared memory pool, thereby achieving physical isolation between the planning agent, the reasoning agent and the original retrieval noise.

[0098] The multi-agent system inputs the raw search data returned by the search engine into the filtering agent. The filtering agent acts as an information gatekeeper, filtering the noisy raw search data, extracting relevant text fragments that can support the answer to the user's question, and writing the extracted relevant text fragments into a shared memory pool.

[0099] Among them, there are two capability building paths for the filtering agent: The first method can rely on preset prompts to extract zero-sample / few-sample information, guiding the filtering agent to screen and extract effective text fragments that are highly related to the search terms output by the planning agent.

[0100] The second approach involves leveraging the offline reinforcement learning process of a multi-agent system to continuously optimize the information extraction and noise discrimination capabilities of the filtering agent. The reinforcement learning scheme will be explained in detail later and will not be elaborated here.

[0101] For example, see [link to previous article] Figure 1After the search engine returns the first round of raw search data, the filtering agent receives the raw search data and loads preset prompts: extract factual fragments directly related to the search term: 1984 (the birth and death years of the novel's author) from the following web page text; remove irrelevant content such as movie information, novel plots, and book reviews, and only retain valid information containing the character's name, birth, and death dates, discarding irrelevant text directly.

[0102] Guided by the aforementioned preset prompts, the filtering agent completes the screening and identification: DocA contains key facts about the birth and death of the target person, from which relevant text fragments are extracted: George Orwell (1903–1950); DocB and DocC are determined to be irrelevant noise content such as film and television information and novel plots and are directly discarded; then the extracted relevant text fragments are stored in the shared memory pool.

[0103] After the second round of searching is completed, the search engine returns the corresponding original webpage. The filtering agent reuses the preset prompts mentioned above to remove irrelevant content such as novel plots and derivative film and television introductions from a new batch of original webpages, extracts relevant text fragments, such as A. Conan Doyle (Born: 1859, Died: 1930), and appends them to the shared memory pool.

[0104] In this way, the filtering agent cleans the massive amount of raw retrieval data, filters out irrelevant and redundant text, stores only high-value fact fragments in the shared memory pool, blocks the transmission of noise from the original web pages to subsequent links, reduces the information load of the planning agent and the reasoning agent at the architectural level, alleviates the cognitive overload caused by long contexts, and reduces the probability of the model generating illusions.

[0105] When the amount of data in the shared memory pool exceeds a threshold, the multi-agent system can perform compression processing on the data in the shared memory pool.

[0106] For example, if the length of the text accumulated in the shared memory pool exceeds a threshold (e.g., 4096 tokens) after multiple rounds of retrieval, the multi-agent system can invoke a pre-trained semantic compression model to condense the data in the shared memory pool into atomic facts.

[0107] As another example, if multiple rounds of retrieval cause the text length in the shared memory pool to exceed the limit, the multi-agent system can also compress the data through summary extraction.

[0108] The abstract extraction can be implemented in various ways, such as text truncation (extracting part of the text content), abstract extraction based on the term frequency-inverse document frequency (TF-IDF) algorithm, and abstract extraction based on the BM25 relevance algorithm.

[0109] Through the above compression process, the context size of the shared memory pool can be controlled, avoiding the continuous accumulation of text in multiple rounds of retrieval that could cause word overflow, while fully preserving core factual information and continuously providing high-density, low-noise observation data for planning and reasoning agents.

[0110] Step S110: The reasoning agent generates response information corresponding to the user's query information based on the data stored in the shared memory pool.

[0111] Adhering to the framework's information isolation constraints, the reasoning agent can read user query information and filtered, compressed valid data from the shared memory pool, but cannot access the original search documents output by the search engine, filtered and discarded noisy text, or historical failed search processes. Based on clean and noise-free factual evidence, the reasoning agent logically deduces and generates the final response information corresponding to the user query.

[0112] When one of the following triggering conditions is met, the inference agent is invoked to generate response information corresponding to the user's query information based on the data stored in the shared memory pool.

[0113] (1) The planning agent determines that there is sufficient information and stops generating search terms; (2) When the data in the shared memory pool reaches the maximum storage capacity, the system terminates the multi-round retrieval loop.

[0114] In this way, on the one hand, relying on the information completeness judgment of the planning agent, reasoning is initiated in a timely manner when there is sufficient information, avoiding unnecessary additional searches and saving computing power and token overhead; on the other hand, setting the maximum capacity of the shared memory pool as a termination condition prevents infinite continuous retrieval, avoids the infinite accumulation of lexical units that may cause system anomalies, and ensures the stability of system operation.

[0115] For example, such as Figure 1 As shown, after multiple rounds of retrieval: the planning agent completes two rounds of search and stores two relevant text fragments in the shared memory pool: George Orwell (1903–1950) and A. Conan Doyle (Born: 1859, Died: 1930).

[0116] Once the planning agent determines that the evidence is complete based on comprehensive information, it stops generating search terms and triggers the inference agent to start.

[0117] The reasoning agent reads two sets of birth and death information from the user query and the shared memory pool, and performs calculations and comparisons: George Orwell's lifespan: 1950 - 1903 = 47 years; A. Conan Doyle's lifespan: 1930 - 1859 = 71 years.

[0118] The final response message is: A.Conan Doyle.

[0119] In this way, the reasoning agent cannot access the raw search data mixed with irrelevant content, and can only make inferences based on the high-density factual evidence after filtering and extraction. It will not be misled by search noise, effectively alleviate the cognitive overload of long context, and reduce factual errors and model illusions.

[0120] In summary, the multi-agent system-based search method provided in one or more embodiments of this specification relies on the hard decoupling of roles between the planning agent, filtering agent, and reasoning agent. This allows the planning agent and reasoning agent to observe only high-value information in the shared memory pool after filtering and extraction, physically shielding them from original retrieval noise and the history of search failures. Each agent performs its dedicated task within a clean information field. This effectively avoids attentional distraction and loss of reasoning chains caused by long contexts, alleviates cognitive overload problems, and enables the model to maintain accurate reasoning ability even when faced with a large amount of noise generated by multiple rounds of retrieval, thereby significantly reducing the probability of factual illusions.

[0121] It should be noted that the effectiveness of the search method based on multi-agent systems described above depends on the actual capabilities of the multi-agent system itself, which need to be acquired through training the multi-agent system.

[0122] However, multi-agent training frameworks in related technologies generally suffer from the problem of credit allocation, which easily leads to "blame-shifting." When the final response is incorrect, the system cannot accurately pinpoint the responsible link and finds it difficult to distinguish whether the root cause is an inaccurate retrieval strategy of the planning agent, a missed key information by the filtering agent, or a deviation in the reasoning logic of the inference agent. This problem will lead to optimization signal oscillations, convergence difficulties, and even training failure during multi-agent joint training.

[0123] To address the aforementioned problem of the difficulty in joint training of multiple agents, one or more embodiments of this specification also provide a reinforcement learning method for multi-agent systems. This method is applicable not only to multi-agent systems including planning agents, filtering agents, and reasoning agents, but also to any scenario requiring joint training of multiple agents.

[0124] The following provides an exemplary description of a reinforcement learning method for a multi-agent system provided in one or more embodiments of this specification.

[0125] Figure 3 This is a flowchart illustrating a reinforcement learning method for a multi-agent system provided in one or more embodiments of this specification.

[0126] like Figure 3As shown, one or more embodiments of this specification provide a reinforcement learning method for a multi-agent system, which may include the following steps: Step S202: Obtain training data.

[0127] The training data acquired in this step consists of pairs of query information and standard response information. The query information is a complex, multi-hop question input by the user, and the standard response information is the correct answer to that question. During training, the standard response information serves as a supervision signal, used to calculate the team's reward (by comparing the model's predicted response information with the standard response information), and also provides a reference for evaluating the large model's performance in calculating the role-specific rewards for each agent.

[0128] The following is an example of a pair of training data: Query information: What were the main research achievements of the 2024 Nobel Prize laureates in Physics? In which fields have these achievements been applied? Standard response information: The 2024 Nobel Prize in Physics was awarded to John J. Hopfield and Geoffrey E. Hinton for their fundamental discoveries and inventions in the fields of artificial neural networks and machine learning. Hopfield's main achievement was the invention of the Hopfield network, which achieved associative memory through the principle of energy minimization, laying the theoretical foundation for information storage and retrieval in neural networks. Hinton's main achievements were the invention of the Boltzmann machine and the generalization of the backpropagation algorithm, which profoundly influenced the development of modern deep learning. Their research results have been widely applied in many fields, including but not limited to: computer vision (such as image recognition and autonomous driving), natural language processing (such as large language models and machine translation), medical diagnosis (such as medical image analysis and drug discovery), and speech recognition. In this example, the query is a multi-hop complexity problem, requiring the agent to first search for the names of the award winners, and then search for their respective research achievements and application areas. The standard response information provides detailed and accurate reference answers, which can be used for literal matching and semantic comparison with the model's prediction results.

[0129] Step S204: Jointly train the multi-agent system based on the training data.

[0130] In this context, a multi-agent system comprises multiple cooperating agents, with each agent performing a different task.

[0131] For example, a multi-agent system may include a planning agent, a filtering agent, and an inference agent. The planning agent generates search terms based on the input query information, which the search engine then uses to retrieve the original retrieval data corresponding to the search terms. The filtering agent extracts relevant text fragments from the original retrieval data and stores these fragments in a shared memory pool. The inference agent generates corresponding predicted response information based on the data stored in the shared memory pool. For details, please refer to the description of the multi-agent system in the embodiments of the search method based on the multi-agent system; it will not be repeated here.

[0132] In one embodiment, such as Figure 4 As shown, the joint training process may include the following steps: Step S2041: For the same query information, sample multiple different trajectories.

[0133] For the same query information in the training data, multiple different trajectories are obtained by executing the complete inference service process multiple times based on the current multi-agent system (for example, the inference service process is the process corresponding to the search method based on the multi-agent system mentioned above).

[0134] Each trajectory can record the complete collaborative sequence from receiving a query to outputting a predicted response. For example, it may include: (1) Plan each step of the agent's decision (generated search terms or stop instructions); (2) The original search data returned by the search engine (but this part is not directly involved in the reward calculation); (3) The filtering agent extracts relevant text fragments from the original retrieval data and stores them in the shared memory pool; (4) The prediction response information finally generated by the reasoning agent.

[0135] Because the agent exhibits a degree of randomness in generating search terms and extracting relevant text fragments, even when faced with the same query, it may sample multiple trajectories with different search paths, filtering results, and final answers. For the same query, the number of sampled trajectories is a preset constant G (e.g., G=3, 5, 8, etc.).

[0136] In this way, multiple trajectories provide different collaborative performance samples for the same query. In subsequent steps, the performance of each agent can be compared horizontally within the group, rather than relying on reward thresholds that are difficult to set fairly.

[0137] Step S2042: For each trajectory, calculate the mixed reward corresponding to each agent in the multi-agent system.

[0138] This step calculates the mixed reward for each agent in the multi-agent system (e.g., planning agent, filtering agent, and reasoning agent) for each trajectory sampled in step S2041.

[0139] In one or more embodiments of this specification, the final mixed reward for each agent can be the sum or weighted sum of the following three rewards.

[0140] (1) Format reward: Used to constrain the output format of each agent to conform to the preset communication protocol specifications. If the format label of the agent's output is incorrect or unparseable, a negative penalty (such as -1 point) is given; if the format is correct, a base score or zero score is given. In the same trajectory, all agents share the same format reward.

[0141] (2) Team result reward: used to measure the final output quality of the entire trajectory. All agents share the same team result reward within the same trajectory.

[0142] For example, the predicted response information generated by the reasoning agent can be compared with the standard response information, and the team result reward can be obtained by weighting the literal matching degree (such as F1Score) and the semantic equivalence score based on the large evaluation model.

[0143] (3) Role-specific rewards: used to characterize the local contribution of the corresponding agent in the collaboration process.

[0144] For example, an independent large-scale evaluation model can be invoked to score each agent based on its own input and output information.

[0145] The following examples illustrate the role-specific reward calculation for planning agents, filtering agents, and reasoning agents.

[0146] 1. For the calculation of role-specific rewards for planning agents, the evaluation model receives the following information for comprehensive evaluation: (1) User query information; (2) Standard response information; (3) Plan the actual inputs of the intelligent agent (user queries, historical search content, and data in the current shared memory pool); (4) Plan the actual output of the intelligent agent (generated search terms or stop instructions).

[0147] The evaluation of the large model can begin by inferring, based on query information and standard response information, how the "ideal planning agent should make decisions" at the current collaborative stage (e.g., which key concepts should be searched and which irrelevant directions should be avoided), and then compare the actual behavior of the planning agent with this ideal behavior.

[0148] Specific scoring dimensions may include: (1) Whether search terms highly relevant to the query were generated; (2) Whether repeated searches for already obtained information were avoided; (3) Whether the generation of search content irrelevant to the task was avoided; (4) Whether the stop command is output in a timely manner when there is sufficient information.

[0149] For example, if a planning agent generates the search term "Hopfield network principle" again after having obtained detailed information about "Hopfield network", the evaluation model will identify this behavior as a redundant search and give it a lower score in this dimension.

[0150] 2. For the role-specific reward calculation of the filtering agent, the evaluation model receives the following information for comprehensive evaluation: (1) User query information; (2) Standard response information; (3) Filter the actual input of the intelligent agent (the raw search data returned by the search engine); (4) Filter the actual output of the agent (extract and store relevant text fragments in the shared memory pool).

[0151] The evaluation model first infers, based on query information and standard response information, "which relevant text fragments the ideal filtering agent should extract from the original retrieval data and which irrelevant information should be removed", and then compares the actual extraction results of the filtering agent with the ideal results.

[0152] Specific scoring dimensions may include: (1) Whether the key facts related to the standard answer were accurately extracted; (2) Whether the necessary contextual information has been fully preserved; (3) Whether it effectively eliminated noisy text such as advertisements, navigation bars, and irrelevant paragraphs; (4) Whether there is any omission or excessive truncation of key information.

[0153] For example, if the original search data contains the key fact that "the 2024 Nobel Prize in Physics was awarded to John Hopfield and Jeffrey Hinton", but the filtering agent only extracts "Nobel Prize in Physics" and omits the names of the winners, the evaluation model will determine that there is a missing key information and give it a lower score.

[0154] 3. For role-specific reward calculation of the reasoning agent, the evaluation model receives the following information for comprehensive evaluation: (1) User query information; (2) Standard response information; (3) The actual input of the reasoning agent (all text fragments that are finally stored in the shared memory pool after planning and filtering). (4) The actual output of the reasoning agent (the generated prediction response information).

[0155] The evaluation model first compares the predicted response information with the standard response information to determine the correctness and completeness of the answer. Based on this, it further analyzes the source of the error: if the predicted response is incorrect, the evaluation model can backtrack to the data in the shared memory pool to determine whether the error is due to the shared memory pool containing the correct answer but the reasoning agent failing to integrate it correctly (attributed to the reasoning agent's own insufficient capabilities), or whether the shared memory pool has lacked key information from the beginning (attributed to errors in upstream planning or filtering).

[0156] For example, if the standard answer is "This achievement is applied to computer vision, natural language processing, and medical diagnosis," while the predicted response only mentions "computer vision and natural language processing," the evaluation model will check if there is text related to "medical diagnosis" in the shared memory pool. If it exists, the omission is attributed to the reasoning agent; if it does not exist, it is determined that the reasoning agent has done its best given the information, and the omission should not be its responsibility.

[0157] In this way, through the aforementioned role-based independent evaluation mechanism, fair and fine-grained role-specific rewards can be generated for each agent in each trajectory. These rewards are not solely influenced by the global outcome reward, ensuring that even if the final answer is incorrect, agents that perform well in the preliminary stages can still receive positive incentives. This effectively breaks the credit allocation dilemma under the traditional global reward mechanism and provides reliable optimization signals for stable joint training of multiple agents.

[0158] For example, taking trajectory 1 as an example, the mixed rewards corresponding to each agent in the multi-agent system are shown in Table 1.

[0159] Table 1

[0160] As shown in Table 1, the planning agent and the filtering agent performed well, achieving high mixed rewards. Although the reasoning agent had flaws, its score did not fall into deep negative values ​​due to the positive pull of the team outcome reward. Conversely, if the planning agent searched a large amount of irrelevant content in another trajectory, even if the reasoning agent ultimately missed application details, its mixed reward would be significantly lower than that of this trajectory due to the lower role-specific reward. This refined reward structure allows the system to distinguish between two different scenarios: "downstream failure due to upstream error" and "upstream effort but downstream error," effectively avoiding confusion in credit allocation.

[0161] Step S2143: Using the group policy optimization algorithm, calculate the advantage value of each agent in each trajectory based on the mixed reward.

[0162] This step, based on the mixed reward of each agent in each trajectory calculated in step S2042, calculates the advantage value of each agent in its respective trajectory by normalizing it to a role-specific group.

[0163] In one possible implementation, the advantage value can be calculated using the following steps: 1. Group the mixed rewards corresponding to agents of the same role type from multiple trajectories sampled for the same query information into a group to obtain multiple role groups.

[0164] For example, taking three trajectories sampled from the same query information, the mixed rewards of all planning agents are grouped into a planning agent group, the mixed rewards of all filtering agents are grouped into a filter group, and the mixed rewards of all reasoning agents are grouped into a reasoner group. Each group contains three scores, corresponding to the performance of that role in the three trajectories. The mixed rewards of each agent in the three trajectories are shown in Table 2.

[0165] Table 2

[0166] 2. Calculate the mean and standard deviation of the mixed rewards within each character group.

[0167] Continuing with Table 2 as an example, the planning agents are grouped as follows: scores are {1.2, 0.6, 0.3}, with a mean μ. P =0.70 standard deviation σ P ≈0.37. Filterer group: scores {1.1, 0.9, 0.4}, mean μ F =0.80, standard deviation σ F ≈0.29. Reasoners were grouped into groups with scores of {0.8, 0.5, 0.2}, with a mean of μ. A =0.50, standard deviation σ A ≈0.24.

[0168] 3. Based on the mixed reward and the mean and standard deviation of the group to which the mixed reward belongs, the advantage value of the corresponding agent in each trajectory is calculated.

[0169] For example, the advantage value of the agent in each trajectory can be calculated using the following relation (1): = Relation (1) in, Let represent the advantage value of agent u in the i-th trajectory. Let represent the mixed reward of agent u in the i-th trajectory. and Let these represent the mean and variance of agent u in its role group, respectively. It is a constant.

[0170] For example, the advantage values ​​of each agent in each trajectory are shown in Table 3.

[0171] Table 3

[0172] Referring to Table 3, the advantage values ​​of all three agents in Trajectory 1 are positive (+1.35, +1.03, +1.25), indicating that Trajectory 1 performs better than the average across all three attempts at this query, and its behavior will be positively encouraged by the optimizer. In Trajectory 3, the advantage values ​​of all three agents are negative (-1.08, -1.38, -1.25), indicating that the overall performance of this trajectory is significantly worse than the average, and its behavior will be suppressed.

[0173] The task difficulty and reward distribution naturally differ among different roles, and directly comparing their original mixed rewards can lead to an imbalance in the training signal. The above implementation calculates the advantage value by normalizing the specific group of roles, transforming the performance of each role into a relative ranking value. This makes the optimization signals across roles comparable on a uniform scale, eliminates the difference in reward scale between roles, and ensures the stability of model updates.

[0174] Step S2144: Update the model parameters corresponding to the multi-agent system based on the advantage value.

[0175] This step uses the dominance value of each agent in each trajectory calculated in step S2143 to update the model parameters corresponding to the multi-agent system. The dominance value reflects the degree to which each agent's behavior in its respective trajectory is superior to the group average level; a positive dominance value indicates that the behavior is worth encouraging, while a negative dominance value indicates that the behavior should be suppressed. Based on this, the optimizer adjusts the model parameters to increase the probability of generating high-dominance behaviors and decrease the probability of generating low-dominance behaviors in the future.

[0176] In one possible implementation, the agents (e.g., planning agent, filtering agent, and inference agent) share the same set of AI model parameters. Therefore, the parameter update process involves merging the advantage values ​​of the three agents along the same trajectory and then applying them collectively to this shared set of parameters. This "merging advantage values, joint update in one step" mechanism allows the AI ​​model to find a universal internal representation that simultaneously satisfies the needs of multiple roles, thus naturally giving rise to self-organizing specialization capabilities.

[0177] For example, for each trajectory, the advantage values ​​of the three agents are combined into a single comprehensive optimization signal. This combination can be achieved through direct summation, weighted summation, or by concatenating the token sequences and calculating the overall probability ratio. For instance, the comprehensive advantage signal for trajectory 1 is +1.35 + 1.03 + 1.25 = +3.63, for trajectory 2 it's +0.07, and for trajectory 3 it's -3.71. Then, the optimizer iterates through the generation sequence of each trajectory. For each token in the sequence, based on its role, it calculates the gradient of the probability ratio using the corresponding advantage value. Specifically, the log probability of generating that token under the model's current policy is multiplied by its corresponding advantage value, which serves as the direction of that token's contribution to parameter updates. Afterward, the gradients of all trajectories and all tokens are summed, and a gradient update is performed on the shared model parameters. After the update, the probability of the model generating similar behaviors to the planning agent, filtering agent, and inference agent in trajectory 1 under similar circumstances will increase, while the probability of generating behaviors similar to those in trajectory 3 will decrease.

[0178] Because the shared parameters are updated jointly by the advantage values ​​of the three roles, the model gradually learns an ability to switch flexibly. For example, when the input received by the model is the exclusive vision of the "planning agent" (user query + historical actions + cleansing the memory pool), it tends to generate accurate search terms; when the input switches to the exclusive vision of the "filtering agent" (raw web page text), it can immediately switch to the mode of extracting key evidence. This context-based role adaptation passively emerges during training in the next joint update of the shared parameters.

[0179] In another implementation, each agent is configured with an independent artificial intelligence model (or an independent LoRA adapter), and the update process involves each agent independently updating its own parameters using its own advantage value, without interfering with each other.

[0180] For example, the planning agent's advantage value sequence {+1.35, -0.27, -1.08} is used to update the planning agent-specific model. The optimizer iterates through all tokens generated by the planning agent in the three trajectories, calculates the gradient based on the advantage value corresponding to each token, and updates the planning agent model parameters based on the gradient. After the update, the planning agent model's ability to generate highly relevant search terms under similar queries is enhanced, while the probability of generating redundant or irrelevant search terms is reduced. This update does not affect the model parameters of the filter and inference agents at all.

[0181] Similarly, updating the filter agent's model using the advantage value sequence {+1.03,+0.34,-1.38} improves its ability to identify and retain high-value factual fragments from the original web pages. The changes in the filter agent's parameters are independent of the planning agent and the inferring agent models. Furthermore, updating the inferring agent's model using the advantage value sequence {+1.25,0.00,-1.25} enhances its ability to generate correct answers based on a shared memory pool. Its parameter updates are independent and do not interfere with the models of other roles.

[0182] In summary, the reinforcement learning method for multi-agent systems provided in one or more embodiments of this specification effectively solves the credit allocation problem commonly found in related technologies during multi-agent joint training by constructing a three-level hybrid reward mechanism that includes format rewards, team result rewards, and role-specific rewards, and by introducing role-specific group normalization for advantage value calculation.

[0183] This method can accurately differentiate the local contributions of different agents. Even if the final response is incorrect, the agent with excellent performance can still receive positive incentives, and the erroneous link can be accurately located, thus making the training signal more accurate and objective. Based on this, the system can achieve stable end-to-end joint training, avoid optimization signal oscillation or training collapse, and thus significantly improve the convergence efficiency and final performance of multi-agent system collaborative optimization.

[0184] The following experimental data verifies the technical effectiveness of the multi-agent system provided in one or more embodiments of this specification in alleviating cognitive overload.

[0185] Experimental Setup: This experiment was conducted on two multi-hop question-answering datasets, HotpotQA and 2WikiMQA. By adjusting the Top-k number of retrieved documents (k values ​​of 3, 5, and 10), an information overload scenario was simulated to evaluate the model's accuracy. Theoretically, increasing the k value can improve the recall rate of relevant evidence, but it also introduces a large amount of irrelevant noise text into the context window, testing the model's ability to distinguish between valid signals and noise. This provides a direct verification of the noise reduction effect of the filtering agent. The single-entity retrieval scheme Search-R1 was selected as the baseline model and compared with our scheme DECOR. The experimental results are shown in Table 4.

[0186] Table 4

[0187] As shown in Table 4, the baseline single-unit solution Search-R1 exhibits a typical inverted U-shaped performance trend. Accuracy peaks at k=5, and performance declines significantly when the number of searches increases to k=10 and noise increases substantially. This phenomenon confirms that excessive search text leads to contextual pollution, causing cognitive overload of the single-unit model and interference from irrelevant information in the reasoning process.

[0188] In stark contrast, the DECOR scheme in this application maintains a continuous upward trend in accuracy as the number of searches (k) increases. This scheme relies on an information isolation mechanism at the architectural level, delegating the noise filtering task to a filtering agent independently, effectively decoupling high evidence recall capability from contextual noise pollution. The reasoning agent can only read extracted and simplified valid facts from the shared memory pool, avoiding contact with massive amounts of raw, mixed search documents. Even with expanded search scale and the introduction of more raw search data, the reasoning agent always bases its deductions on clean, high-value factual information, fully utilizing the valid evidence in more search results without being overwhelmed by noise.

[0189] The following comparative experiments are conducted using multiple publicly available deep retrieval benchmark datasets to verify the comprehensive performance advantages of the multi-agent system provided in one or more embodiments of this specification in deep retrieval question answering tasks.

[0190] Experimental Setup: The experiment selected seven publicly available deep search benchmark datasets: NQ, TriviaQA, HotpotQA, 2WikiMQA, MuSiQue, Bamboogle, and PopQA, covering various task scenarios such as single-hop question answering and multi-hop complex reasoning. Two evaluation metrics were used: the LJ metric, representing the semantic accuracy judged by a large model, measures the semantic correctness of the answer; and the F1 metric, representing the word overlap score, focusing on the degree of text string matching. Mainstream deep search methods such as SimpleDeepSearcher, AutoRefine, CoRAG, R1-Searcher, Search-R1, and o2-searcher were selected as baselines for comparison. Experimental results are as follows: Figure 5 As shown in Table 5.

[0191] like Figure 5 As shown, the DECOR scheme in this application achieves the highest LJ semantic accuracy on all seven datasets, fully demonstrating that the system can stably output semantically correct responses and maintain reliable performance even in scenarios with fluctuating word matching conditions. It also exhibits outstanding robustness for complex multi-hop inference tasks. Taking the typical multi-hop dataset HotpotQA as an example, DECOR significantly outperforms the single-model baseline model Search-R1, while consistently surpassing mainstream deep retrieval schemes such as SimpleDeepSearcher.

[0192] By distinguishing between lexical and semantic metrics, a key advantage can be identified: On datasets such as 2WikiMQA and Bamboogle, single-agent baselines like R1-Searcher and Search-R1 achieve slightly higher F1 scores due to their strict string matching; however, the proposed DECOR model achieves higher LJ semantic judgment accuracy. This phenomenon indicates that single-agent models are more likely to output responses that only overlap with the standard answer in terms of text fragments but have semantic logical flaws; while this multi-agent system, relying on observation isolation and a collaborative architecture, can generate responses with more accurate semantics and more coherent logic.

[0193] Meanwhile, DECOR outperforms solutions like o2-searcher, which rely solely on static computation scaling during inference, across all metrics. This verifies that for deep retrieval scenarios, training specialized multi-agent collaborative strategies yields superior optimization benefits compared to simply statically increasing inference computing power. At its core, this system decouples retrieval planning, information filtering, and logical reasoning tasks and sets observation boundaries. It relies on filtering agents to extract relevant text fragments, avoiding interference from raw retrieval noise in the inference chain, thus ensuring the quality of semantic reasoning in complex question-answering tasks from an architectural perspective.

[0194] It is understood that the above embodiments are merely examples, and modifications can be made to the above embodiments in actual implementation. Those skilled in the art will understand that any modifications to the above embodiments that do not require creative effort fall within the protection scope of one or more embodiments of this specification, and will not be described again in the embodiments.

[0195] Based on the same inventive concept, one or more embodiments of this specification also provide a multi-agent system. The principle of solving the problem by the multi-agent system is similar to that of the aforementioned search method based on the multi-agent system. Therefore, the implementation of the multi-agent system can refer to the implementation of the aforementioned search method based on the multi-agent system, and the repeated parts will not be described again.

[0196] See Figure 1 This specification provides a multi-agent system according to one or more embodiments, comprising: a planning agent for generating search terms based on user query information, for a search engine to search for original retrieval data corresponding to the search terms; a filtering agent for extracting relevant text fragments from the original retrieval data and storing the relevant text fragments in a shared memory pool; and a reasoning agent for generating response information corresponding to the user query information based on the data stored in the shared memory pool.

[0197] Based on the same inventive concept, one or more embodiments of this specification also provide a search device based on a multi-agent system. Since the principle of the problem solved by the search device based on a multi-agent system is similar to that of the aforementioned search method based on a multi-agent system, the implementation of the search device based on a multi-agent system can refer to the implementation of the aforementioned search method based on a multi-agent system, and the repeated parts will not be described again.

[0198] See Figure 6 , Figure 6 This is a structural block diagram of a search device based on a multi-agent system, provided for one or more embodiments of this specification. Figure 6 As shown, the search device 300 based on a multi-agent system may include: an acquisition module 301, a planning module 302, a search module 303, a filtering module 304, and an inference module 305. Among them, The acquisition module 301 is used to acquire user query information and initialize the shared memory pool; Planning module 302 is used to call the planning agent to generate search terms based on user query information; Search module 303 is used to call the search engine to search for the original retrieval data corresponding to the search terms; The filtering module 304 is used to call the filtering agent to extract relevant text fragments from the original retrieval data and store the relevant text fragments in the shared memory pool; The inference module 305 calls the inference agent to generate response information corresponding to the user's query information based on the data stored in the shared memory pool.

[0199] In one possible implementation, the planning module 302 is used to generate a first search term based on the user query information when the planning agent is called in the first round; and to generate a second search term based on the user query information, historical search records, and data stored in the shared memory pool when the planning agent is called in the Nth round, where N is an integer greater than 1, and the historical search records include the search terms generated by the planning agent in each round before the Nth round for the user query information.

[0200] In one possible implementation, the planning module 302 is further configured to stop generating search terms when it is determined that the search stopping conditions are met, based on user query information, historical search records, and data stored in the shared memory pool.

[0201] In one possible implementation, the filtering module 304 is used to guide the filtering agent to extract relevant text fragments related to the search term based on preset prompt words.

[0202] One possible implementation also includes a compression module for compressing data in the shared memory pool when the amount of data in the shared memory pool exceeds a threshold.

[0203] In one possible implementation, a compression module is used to compress the data in the shared memory pool into atomic facts using a pre-trained semantic compression model when the amount of data in the shared memory pool exceeds a threshold, or to compress the data by extracting a summary from the data in the shared memory pool.

[0204] In one possible implementation, the inference module 305 is used to call the inference agent to generate response information corresponding to the user query information based on the data stored in the shared memory pool when the planning agent stops generating search terms or the shared memory pool reaches its maximum storage capacity.

[0205] In one possible implementation, at least two of the planning agent, filtering agent, and reasoning agent share the same artificial intelligence model or adapter; or, the planning agent, filtering agent, and reasoning agent are each configured with an independent artificial intelligence model or adapter.

[0206] Based on the same inventive concept, one or more embodiments of this specification also provide a reinforcement learning device for a multi-agent system. Since the principle of the problem solved by the reinforcement learning device for a multi-agent system is similar to that of the aforementioned reinforcement learning method for a multi-agent system, the implementation of the reinforcement learning device for a multi-agent system can refer to the implementation of the aforementioned reinforcement learning method for a multi-agent system, and the repeated parts will not be described again.

[0207] See Figure 7 , Figure 7 This is a structural block diagram of a reinforcement learning device for a multi-agent system provided in one or more embodiments of this specification. Figure 7 As shown, the reinforcement learning device 400 for the multi-agent system may include: an acquisition module 401 and a training module 402. Wherein, The acquisition module 401 is used to acquire training data, which includes paired query information and standard response information; Training module 402 is used to jointly train a multi-agent system based on training data. The multi-agent system includes multiple cooperating agents, and different agents are used to perform different tasks. The training module 402 is specifically used for: sampling multiple different trajectories for the same query information; for each trajectory, calculating the mixed reward corresponding to each agent in the multi-agent system, the mixed reward including format reward, team result reward and role-specific reward, the role-specific reward is used to represent the local contribution of the corresponding agent; using the group policy optimization algorithm, calculating the advantage value of each agent in each trajectory based on the mixed reward; and updating the model parameters corresponding to the multi-agent system based on the advantage value.

[0208] In one possible implementation, the multi-agent system includes a planning agent, a filtering agent, and an inference agent; wherein, the planning agent is used to generate search terms based on the input query information, so that the search engine can search for the original retrieval data corresponding to the search terms; the filtering agent is used to extract relevant text fragments from the original retrieval data and store the relevant text fragments in a shared memory pool; and the inference agent is used to generate corresponding predicted response information based on the data stored in the shared memory pool.

[0209] In one possible implementation, the training module 402 is specifically used to calculate the role-specific reward for each agent based on the input and output information of each agent in the multi-agent system using the evaluation model.

[0210] In one possible implementation, the training module 402 is specifically used to group the mixed rewards corresponding to agents of the same role type in multiple trajectories sampled for the same query information into multiple role groups; calculate the mean and standard deviation of the mixed rewards in each role group; and calculate the advantage value of the corresponding agent in each trajectory based on the mixed rewards and the mean and standard deviation of the group to which the mixed rewards belong.

[0211] In one possible implementation, the training module 402 is specifically used to calculate the advantage value of the agent in each trajectory using the following relation: =

[0212] in, Let represent the advantage value of agent u in the i-th trajectory. Let represent the mixed reward of agent u in the i-th trajectory. and Let these represent the mean and variance of agent u in its role group, respectively. It is a constant.

[0213] In one possible implementation, the agents in the multi-agent system share the same artificial intelligence model or adapter; the training module is specifically used to merge the advantage values ​​of each agent in the multi-agent system, and jointly update the parameters of the shared artificial intelligence model or adapter based on the merged advantage values.

[0214] In one possible implementation, each agent in the multi-agent system is configured with an independent artificial intelligence model or adapter; the training module is specifically used to update the parameters of the corresponding artificial intelligence model or adapter based on the advantage value of each agent.

[0215] See Figure 8 , Figure 8 This is a structural block diagram of an electronic device provided for one or more embodiments of this specification. Figure 8 As shown, the electronic device 500 may include a processor 501 and a memory 502; the memory 502 may be coupled to the processor 501. It is worth noting that... Figure 8 This is an example; other types of structures can also be used to supplement or replace this structure to achieve telecommunications functions or other functions.

[0216] In one possible implementation, the functionality of a multi-agent system, a multi-agent system-based search device 300, or a multi-agent system-based reinforcement learning device 400 can be integrated into the processor 501. The processor 501 can be configured to execute the corresponding methods.

[0217] In another possible implementation, the multi-agent system, the multi-agent system-based search device 300, or the multi-agent system-based reinforcement learning device 400 can be configured separately from the processor 501. For example, the multi-agent system, the multi-agent system-based search device 300, or the multi-agent system-based reinforcement learning device 400 can be configured as a chip connected to the processor 501, and the corresponding method can be implemented through the control of the processor 501.

[0218] Furthermore, in some alternative implementations, the electronic device 500 may also include: a communication module, an input unit, an audio processor, a display, a power supply, etc. It is worth noting that the electronic device 500 is not necessarily required to include these components. Figure 8 All components shown; in addition, the electronic device 500 may also include Figure 8 For components not shown, please refer to existing technologies.

[0219] In some alternative implementations, the processor 501, sometimes also referred to as a controller or operating control, may include a microprocessor or other processor device and / or logic device, which receives input and controls the operation of various components of the electronic device 500.

[0220] The memory 502 may be, for example, one or more of the following: a cache, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable device. It can store the aforementioned information related to the multi-agent system, the multi-agent system-based search device 300, or the multi-agent system-based reinforcement learning device 400, and may also store programs for executing that information. The processor 501 can execute the program stored in the memory 502 to perform information storage or processing, etc.

[0221] An input unit can provide input to the processor 501. This input unit may be, for example, a button or touch input device. A power supply can be used to provide power to the electronic device 500. A display can be used to display images and text, etc. This display may be, for example, an LCD display, but is not limited to this.

[0222] Memory 502 can be a solid-state memory, such as read-only memory (ROM), random access memory (RAM), SIM card, etc. It can also be a memory that retains information even when power is off, can be selectively erased, and contains more data; examples of this type of memory are sometimes referred to as EPROM, etc. Memory 502 can also be some other type of device. Memory 502 includes buffer memory (sometimes referred to as a buffer). Memory 502 may include an application / function storage unit for storing application programs and function programs or processes for executing the operation of electronic device 500 via processor 501.

[0223] The memory 502 may also include a data storage unit for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit of the memory 502 may include various drivers for the computer device for communication functions and / or for performing other functions of the computer device (such as messaging applications, address book applications, etc.).

[0224] The communication module is a transmitter / receiver that sends and receives signals via an antenna. The communication module (transmitter / receiver) is coupled to the processor 501 to provide input signals and receive output signals, which is the same as in a conventional mobile communication terminal.

[0225] Based on different communication technologies, multiple communication modules can be configured in the same computer device, such as cellular network modules, Bluetooth modules, and / or wireless LAN modules. The communication modules (transmitters / receivers) are also coupled to speakers and microphones via an audio processor to provide audio output through the speakers and receive audio input from the microphones, thereby enabling typical telecommunications functions. The audio processor may include any suitable buffer, decoder, amplifier, etc. Additionally, the audio processor is coupled to processor 501, enabling on-device recording via the microphone and on-device playback of stored sound via the speakers.

[0226] This specification also provides a computer program product in one or more embodiments, the computer program product comprising: a computer program or instructions that, when run on a computer, cause the computer to perform the method of any one of the method embodiments.

[0227] This specification also provides a computer storage medium in one or more embodiments that stores a computer program or instructions that, when executed on a computer, cause the computer to perform the method of any one of the method embodiments.

[0228] While one or more embodiments of this specification provide the operational steps of the methods described in the embodiments or flowcharts, more or fewer operational steps may be included based on conventional or non-inventive labor. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only possible execution order. In actual device or client product execution, the methods shown in the embodiments or drawings can be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment).

[0229] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, apparatus (system), or computer program product. Therefore, one or more embodiments of this specification can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0230] This specification describes one or more embodiments of a method, apparatus (system), and computer program product according to one or more embodiments of this specification with reference to flowchart illustrations and / or block diagrams. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0231] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0232] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0233] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and system embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0234] In this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, without necessarily requiring or implying any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Those skilled in the art will understand the specific meaning of the above terms in one or more embodiments of this specification, depending on the specific circumstances.

[0235] It should be noted that, unless otherwise specified, one or more embodiments and features thereof in this specification can be combined with each other. This specification is not limited to any single aspect, nor to any single embodiment, nor to any combination and / or substitution of such aspects and / or embodiments. Furthermore, each aspect and / or embodiment of one or more embodiments of this specification can be used alone or in combination with one or more other aspects and / or embodiments thereof.

[0236] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of one or more embodiments of this specification, and are not intended to limit them. Although one or more embodiments of this specification have been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of one or more embodiments of this specification, and they should all be covered within the scope of the claims and the specification of one or more embodiments of this specification.

[0237] The foregoing description of one or more embodiments of this specification has been provided in conjunction with optional implementation methods. However, these embodiments are merely exemplary and illustrative. Various substitutions and modifications can be made to the one or more embodiments of this specification, all of which fall within the protection scope of the one or more embodiments of this specification.

Claims

1. A search method based on a multi-agent system, characterized in that, The multi-agent system includes a planning agent, a filtering agent, and a reasoning agent; the method includes: Obtain user query information and initialize the shared memory pool; The planning agent is invoked to generate search terms based on the user query information; Call the search engine to retrieve the original search data corresponding to the search terms; The filtering agent is invoked to extract relevant text fragments from the original retrieval data and store the relevant text fragments in the shared memory pool; The reasoning agent is invoked to generate response information corresponding to the user query information based on the data stored in the shared memory pool.

2. The method as described in claim 1, characterized in that, The planning agent is invoked to generate search terms based on the user query information, including: In the first round of invocation of the planning agent, a first search term is generated based on the user query information; In the Nth round of calling the planning agent, a second search term is generated based on the user query information, historical search records, and data stored in the shared memory pool, wherein N is an integer greater than 1, and the historical search records include search terms generated by the planning agent in each round prior to the Nth round in response to the user query information.

3. The method as described in claim 2, characterized in that, The method further includes: If, based on the user query information, the historical search records, and the data stored in the shared memory pool, the search stopping condition is met, the generation of search terms is stopped.

4. The method as described in claim 1, characterized in that, The filtering agent is invoked to extract relevant text fragments from the original retrieval data, including: Based on preset prompts, the filtering agent is guided to extract relevant text fragments related to the search term.

5. The method as described in claim 1, characterized in that, The method further includes: If the amount of data in the shared memory pool exceeds a threshold, the data in the shared memory pool is compressed.

6. The method as described in claim 5, characterized in that, When the amount of data in the shared memory pool exceeds a threshold, data compression processing is performed on the shared memory pool, including: If the amount of data in the shared memory pool exceeds a threshold, the data in the shared memory pool is compressed into atomic facts using a pre-trained semantic compression model, or the data in the shared memory pool is compressed by performing a summary extraction.

7. The method as described in claim 5, characterized in that, The reasoning agent is invoked to generate response information corresponding to the user query information based on the data stored in the shared memory pool, including: If the planning agent stops generating search terms or the shared memory pool reaches its maximum storage capacity, the reasoning agent is invoked to generate response information corresponding to the user query information based on the data stored in the shared memory pool.

8. The method according to any one of claims 1-7, characterized in that, At least two of the planning agent, the filtering agent, and the reasoning agent share the same artificial intelligence model or adapter; or... The planning agent, the filtering agent, and the reasoning agent are each configured with an independent artificial intelligence model or adapter.

9. A reinforcement learning method for a multi-agent system, characterized in that, The method includes: Acquire training data, which includes pairs of query information and standard response information; The multi-agent system is jointly trained based on the training data. The multi-agent system includes multiple cooperating agents, and different agents are used to perform different tasks. The joint training process includes the following steps: For the same query information, sample multiple different trajectories; For each trajectory, the mixed reward corresponding to each agent in the multi-agent system is calculated. The mixed reward includes format reward, team result reward and role-specific reward. The role-specific reward is used to characterize the local contribution of the corresponding agent. Using a group policy optimization algorithm, the advantage value of each agent in each trajectory is calculated based on the hybrid reward. Based on the aforementioned advantage value, the model parameters corresponding to the multi-agent system are updated.

10. The method as described in claim 9, characterized in that, The multi-agent system includes a planning agent, a filtering agent, and a reasoning agent; The planning agent is used to generate search terms based on the input query information, so that the search engine can search the original retrieval data corresponding to the search terms; the filtering agent is used to extract relevant text fragments from the original retrieval data and store the relevant text fragments in a shared memory pool; the reasoning agent is used to generate corresponding predicted response information based on the data stored in the shared memory pool.

11. The method as described in claim 9 or 10, characterized in that, Calculating the role-specific reward for each agent in the multi-agent system includes: Using a large evaluation model, based on the input and output information of each agent in the multi-agent system, the role-specific reward for each agent is calculated.

12. The method as described in claim 9 or 10, characterized in that, Using a group policy optimization algorithm, the advantage value of each agent in each trajectory is calculated based on the hybrid reward, including: Grouping the mixed rewards corresponding to agents of the same role type into a group from multiple trajectories sampled for the same query information, we obtain multiple role groups. Calculate the mean and standard deviation of the mixed rewards within each role group; Based on the mixed reward and the mean and standard deviation of the group to which the mixed reward belongs, the advantage value of the corresponding agent in each trajectory is calculated.

13. The method as described in claim 12, characterized in that, Based on the hybrid reward and the mean and standard deviation of the group to which the hybrid reward belongs, the dominance value of the corresponding agent in each trajectory is calculated, including: The advantage value of the agent in each trajectory is calculated using the following formula: = in, Let represent the advantage value of agent u in the i-th trajectory. Let represent the mixed reward of agent u in the i-th trajectory. and Let these represent the mean and variance of agent u in its respective role group, respectively. It is a constant.

14. The method as described in claim 9 or 10, characterized in that, In the multi-agent system, each agent shares the same artificial intelligence model or adapter; wherein, updating the model parameters corresponding to the multi-agent system based on the advantage value includes: The advantage values ​​of each agent in the multi-agent system are merged, and the parameters of the shared artificial intelligence model or adapter are updated based on the merged advantage values.

15. The method as described in claim 9 or 10, characterized in that, Each agent in the multi-agent system is configured with an independent artificial intelligence model or adapter. The process of updating the model parameters of the multi-agent system based on the advantage value includes: Based on the advantage value of each agent, update the parameters of the corresponding AI model or adapter.

16. A multi-agent system, characterized in that, The system includes: A planning agent is used to generate search terms based on user query information, so that the search engine can search for the original retrieval data corresponding to the search terms; A filtering agent is used to extract relevant text fragments from the original retrieval data and store the relevant text fragments in a shared memory pool; An inference agent is used to generate response information corresponding to the user query information based on the data stored in the shared memory pool.

17. A search device based on a multi-agent system, characterized in that, The multi-agent system includes a planning agent, a filtering agent, and a reasoning agent; the device includes: The acquisition module is used to obtain user query information and initialize the shared memory pool; The planning module is used to invoke the planning agent to generate search terms based on the user query information; The search module is used to call the search engine to search for the original retrieval data corresponding to the search terms. The filtering module is used to call the filtering agent to extract relevant text fragments from the original retrieval data and store the relevant text fragments in the shared memory pool; The reasoning module calls the reasoning agent to generate response information corresponding to the user query information based on the data stored in the shared memory pool.

18. A reinforcement learning device for a multi-agent system, characterized in that, include: The acquisition module is used to acquire training data, which includes paired query information and standard response information; A training module is used to jointly train a multi-agent system based on the training data. The multi-agent system includes multiple cooperating agents, and different agents are used to perform different tasks. The training module is specifically used for: For the same query information, sample multiple different trajectories; For each trajectory, the mixed reward corresponding to each agent in the multi-agent system is calculated. The mixed reward includes format reward, team result reward and role-specific reward. The role-specific reward is used to characterize the local contribution of the corresponding agent. Using a group policy optimization algorithm, the advantage value of each agent in each trajectory is calculated based on the hybrid reward. Based on the aforementioned advantage value, the model parameters corresponding to the multi-agent system are updated.

19. An electronic device, characterized in that, The electronic device includes: Memory, used to store computer program products; A processor is configured to execute a computer program product stored in the memory, wherein, when the computer program product is executed, it implements the method described in any one of claims 1-15.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed, implement the method described in any one of claims 1-15.

Images