Data processing method and related system

Abstract

A data processing method, comprising: acquiring a question; on the basis of the question and by means of interaction between a plurality of agents, obtaining a plurality of pieces of first interaction information, the plurality of pieces of first interaction information comprising a first reply to the question; when the first reply is not a correct reply to the question, on the basis of the plurality of pieces of first interaction information and by means of a reflector, determining reason information corresponding to each of the agents, the reason information being a reason for obtaining the incorrect reply; and, on the basis of the question and the reason information corresponding to each of the agents and by means of interaction between the plurality of agents, obtaining a second reply to the question. The present application can improve the collaboration capability of multi-agent systems and the accuracy of generated replies, and reduce time overheads of generation processes.

Description

Data processing method and related system

[0001] The present application claims priority to the Chinese patent application No. 202411394315.9, filed on September 30, 2024, and entitled "A data processing method and related system", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence (AI) technology, and in particular to a data processing method and device, a computing device, a chip, a computer readable storage medium, and a computer program product. BACKGROUND

[0003] Large language models (LLMs) encapsulate rich human knowledge in their vast parameters and have achieved good performance in many complex natural language tasks, including question answering. With the powerful language expression and reasoning planning capabilities of large models, autonomous agents based on large models can effectively understand and generate instructions, interact and make decisions in complex environments, and have achieved good results in various downstream tasks. To cope with more complex task scenarios, researchers have proposed building multi-agent systems to divide complex tasks into several subtasks and assign them to multiple agents with different domain skills and professional knowledge.

[0004] The division of agents in a multi-agent system mainly relies on the prior knowledge of researchers, and the accuracy of the generated replies is poor, and the generation time is also longer. Therefore, how to better improve the generation accuracy of replies through multi-agents is a technical problem to be solved. SUMMARY

[0005] The present application provides a data processing method, and a device, a computing device cluster, a computer readable storage medium, and a computer program product corresponding to the above method.

[0006] In a first aspect, the present application provides a data processing method, comprising: obtaining a question; obtaining a plurality of first interaction information through interaction between a plurality of agents according to the question, the plurality of first interaction information comprising a first reply to the question; when the first reply is not a correct answer to the question, determining reason information corresponding to each agent through a reflector according to the plurality of first interaction information, the reason information being a reason for obtaining an incorrect reply; and obtaining a second reply to the question through interaction between the plurality of agents according to the question and the reason information corresponding to each agent.

[0007] The method provided in the embodiments of the present application can reflect, according to interaction information obtained by multiple agents, when cooperation among the multiple agents does not obtain a correct reply, the purpose of the reflection being to determine the reason of each agent why a correct reply is not obtained, and based on the information as priori, participate in the process of determining a reply in the next round of multi-agent cooperation, which can improve the cooperation capability of the multi-agent system, the accuracy of generating a reply, and reduce the time cost of the generation process.

[0008] In a possible implementation, the same reflector is used to determine the reason information corresponding to each agent.

[0009] In the embodiments of the present application, a shared reflector of a multi-agent cooperation system is constructed. Considering the homogeneity among different reflectors, that is, their action spaces (reflection) are consistent, and their optimization objectives are also completely consistent (auxiliary solution of the overall task), the embodiments of the present application propose that all agents share the same reflector, and through carefully designed prompts, the shared reflector can perceive the role information of different agents. This sharing mechanism not only reduces the demand for computing resources, but also makes the model training data more, and the training more stable.

[0010] In a possible implementation, the multiple agents include a target agent; and the second reply to the question is obtained through interaction among the multiple agents according to the question and the reason information corresponding to each agent, including: second interaction information corresponding to each agent is obtained through interaction among the multiple agents according to the question and the reason information corresponding to each agent, and the multiple second interaction information includes the second reply.

[0011] In a possible implementation, the second interaction information corresponding to the target agent is obtained in the following manner:

[0012] The second interaction information of the target agent is obtained through interaction between the target agent and other agents according to the question and the reason information corresponding to the target agent.

[0013] In a possible implementation, the second interaction information corresponding to the target agent is obtained in the following manner:

[0014] The second interaction information of the target agent is obtained through interaction between the target agent and other agents according to the question and the reason information corresponding to the target agent.

[0015] In a possible implementation, the method further includes:

[0016] obtain a label of the cause information corresponding to each of the agents, and a weight corresponding to the cause information; the weight indicates a degree of positive influence of using the reflector on accuracy of the obtained second interaction information compared with not using the reflector; and fine-tune the reflector according to the label and the weight. For example, a loss can be constructed according to the label and the cause information, and then the loss is adjusted by using the weight.

[0017] Embodiments of the present application propose to construct counterfactual rewards as a supervision signal for fine-tuning of individual agents, and use a proximal policy optimization method to fine-tune the reflector of the agent. First, the reflections of all agents are added to the hints of the corresponding decision model to obtain the overall reward. Then, the reflection of each agent is removed in turn, and the process of interacting with the environment is repeated, and the reward obtained is used as the marginal reward. The difference between the overall reward and the marginal reward is used as the contribution evaluation of the individual agent in the cooperation process, and is used as a supervision signal for fine-tuning the reflector of the agent. The reflector is optimized by the proximal policy optimization method, so as to maximize the environment reward.

[0018] In a possible implementation, the agent is a large language model (LLM).

[0019] In a second aspect, the present application provides a data processing apparatus, which comprises:

[0020] a question obtaining module configured to obtain a question;

[0021] a reply determining module configured to obtain a plurality of first interaction information through interaction among a plurality of agents according to the question, the plurality of first interaction information including a first reply to the question; when the first reply is not a correct answer to the hint, determine cause information corresponding to each of the agents through a reflector according to the plurality of first interaction information, the cause information being a reason for obtaining an incorrect reply; and obtain a second reply to the question through interaction among the plurality of agents according to the question and the cause information corresponding to each of the agents.

[0022] wherein the second interaction information corresponding to the target agent is obtained by using the same reflector when determining the cause information corresponding to each of the agents.

[0023] wherein the second interaction information corresponding to the target agent is obtained by using the same reflector when determining the cause information corresponding to each of the agents.

[0024] The reply determining module is specifically configured to:

[0025] According to the question and the cause information corresponding to each of the agents, second interaction information corresponding to each of the agents is obtained through interaction between the agents, and the second interaction information includes the second reply.

[0026] The second interaction information corresponding to the target agent is obtained in the following manner: the second interaction information corresponding to the target agent is obtained in the following manner:

[0027] According to the question and the cause information corresponding to the target agent, second interaction information of the target agent is obtained through interaction between the target agent and other agents.

[0028] The second interaction information corresponding to the target agent is obtained in the following manner: the second interaction information corresponding to the target agent is obtained in the following manner:

[0029] According to the question and the cause information corresponding to the target agent, second interaction information of the target agent is obtained through interaction between the target agent and other agents.

[0030] The second interaction information corresponding to the target agent is obtained in the following manner: the second interaction information corresponding to the target agent is obtained in the following manner:

[0031] The model training module is configured to obtain a label of the cause information corresponding to each of the agents and a weight corresponding to the cause information, and the weight indicates a positive influence degree of the accuracy of the second interaction information obtained by using the reflector compared with not using the reflector.

[0032] The reflector is fine-tuned according to the label and the weight.

[0033] In a possible implementation, the agent is a large language model (LLM).

[0034] In a third aspect, the present application provides a computing device. The computing device includes a memory and a processor; the memory stores code, and the processor is configured to execute the code, when the code is executed, the computing device executes the method of the first aspect or any implementation manner of the first aspect.

[0035] In a fourth aspect, the present application provides a chip. The chip includes at least one processing unit and an interface circuit, the interface circuit is used to provide program instructions or data for the at least one processing unit, and the at least one processing unit is used to execute the program instructions to realize the method of the first aspect or any implementation manner of the first aspect.

[0036] In a fifth aspect, the present application provides a computing device cluster. The computing device cluster comprises at least one computing device, and the at least one computing device comprises at least one processor and at least one memory. The at least one processor and the at least one memory are in communication with each other. The at least one processor is configured to execute instructions stored in the at least one memory, so that the computing device or the computing device cluster performs the method of the first aspect or any implementation manner of the first aspect.

[0037] In a sixth aspect, the present application provides a computer readable storage medium, and the computer readable storage medium stores instructions for instructing a computing device or a computing device cluster to perform the data processing method of the first aspect or any implementation manner of the first aspect.

[0038] In a seventh aspect, the present application provides a computer program product comprising instructions for causing a computing device or a computing device cluster to perform the data processing method of the first aspect or any implementation manner of the first aspect when the computer program product is executed on the computing device or the computing device cluster.

[0039] On the basis of the implementation manners of the above aspects, the present application can be further combined to provide more implementation manners. BRIEF DESCRIPTION OF DRAWINGS

[0040] FIG. 1A is a structural schematic diagram of an artificial intelligence main body framework;

[0041] FIG. 1B and FIG. 2 are schematic diagrams of an application system framework of the present application;

[0042] FIG. 3 is a schematic diagram of an optional hardware structure of a terminal;

[0043] FIG. 4 is a structural schematic diagram of a server;

[0044] FIG. 5 is a schematic diagram of a system architecture of the present application;

[0045] FIG. 6 is a flow of a cloud service;

[0046] FIG. 7 is a flow schematic diagram of a data processing method provided by an embodiment of the present application;

[0047] FIG. 8 to FIG. 9C are schematic diagrams of an application architecture of an embodiment of the present application;

[0048] FIG. 9D is a schematic diagram of a beneficial effect of an embodiment of the present application;

[0049] FIG. 10 is a structural schematic diagram of a data processing apparatus provided by an embodiment of the present application;

[0050] FIG. 11 is a structural schematic diagram of a terminal device provided by an embodiment of the present application;

[0051] FIG. 12 is a schematic diagram of a structure of a server according to an embodiment of the present application;

[0052] FIG. 13 is a schematic diagram of a structure of a chip according to an embodiment of the present application. DETAILED DESCRIPTION

[0053] The embodiments of the present application will be described below in conjunction with the accompanying drawings. The terms used in the embodiments of the present application are only used to explain the embodiments of the present application, and are not intended to limit the present application.

[0054] The embodiments of the present application will be described below in conjunction with the accompanying drawings. The embodiments of the present application are applicable to similar technical problems as the technical solutions provided by the embodiments of the present application are known to those skilled in the art.

[0055] The terms "first", "second", and the like in the description of the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged, and this is only a distinguishing way used in the description of the embodiments of the present application to describe the objects with the same attributes. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, so that the processes, methods, systems, products or equipment containing a series of units do not have to be limited to those units, but can include other units not clearly listed or inherent to these processes, methods, products or equipment.

[0056] The terms "substantially", "about", and the like used in the present application are used as approximate terms, not as degree terms, and are intended to take into account the inherent deviations of measured or calculated values known to those skilled in the art. In addition, the use of "may" in describing the embodiments of the present application means "one or more embodiments". The terms "use", "using", and "used" used in the present application can be considered as synonymous with the terms "utilize", "utilizing", and "utilized", respectively. In addition, the term "exemplary" is intended to refer to an example or an illustration.

[0057] First, the overall workflow of the artificial intelligence system is described, please refer to FIG. 1A, which shows a structural diagram of an artificial intelligence subject framework, and the following describes the artificial intelligence subject framework from two dimensions of "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis). The "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, data undergoes a condensation process of "data-information-knowledge-wisdom". The "IT value chain" reflects the value brought by artificial intelligence to the information technology industry from the underlying infrastructure of human intelligence, information (provision and processing technology implementation) to the industrial ecological process of the system.

[0058] (1) Infrastructure

[0059] The infrastructure provides computing power support for the artificial intelligence system, realizes communication with the outside world, and realizes support through the underlying platform. Communication with the outside world through sensors; computing power is provided by intelligent chips (CPU, NPU, GPU, ASIC, FPGA, etc. Hardware acceleration chips); the underlying platform includes distributed computing framework and network related platform guarantee and support, which can include cloud storage and computing, interconnection network, etc. For example, sensors and external communication acquire data, which are provided to intelligent chips in the distributed computing system provided by the underlying platform for calculation.

[0060] (2) Data

[0061] The data on the upper layer of the infrastructure is used to represent the data source in the field of artificial intelligence. Data involves graphics, images, speech, text, and also involves Internet of Things data of traditional devices, including business data of existing systems and sensing data such as force, displacement, liquid level, temperature, humidity, etc.

[0062] (3) Data processing

[0063] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making, etc.

[0064] Among them, machine learning and deep learning can symbolize and formalize intelligent information modeling, extraction, preprocessing, training, etc.

[0065] Reasoning refers to the process of simulating human intelligent reasoning methods in computers or intelligent systems, using formalized information to perform machine thinking and solve problems according to reasoning control strategies, and the typical function is search and matching.

[0066] Decision-making refers to the process of decision-making after intelligent information reasoning, which usually provides functions such as classification, sorting, prediction, etc.

[0067] (4) General capabilities

[0068] After the data is processed as mentioned above, further based on the result of the data processing, some general capabilities can be formed, such as an algorithm or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0069] (5) Intelligent products and industry applications

[0070] Intelligent products and industry applications refer to the products and applications of artificial intelligence systems in various fields, which are the packaging of the overall solution of artificial intelligence, and realize the application of intelligent information decision-making products. The application fields mainly include intelligent terminals, intelligent transportation, intelligent medical treatment, automatic driving, smart cities, etc.

[0071] First, the application scenario of the present application is introduced. The present application can be but is not limited to an application program (hereinafter referred to as a text generation application program) of a generative artificial intelligence (artificial intelligence generated content, AIGC) function or a cloud service provided by a cloud-side server, etc., which will be introduced next:

[0072] I. Text generation application program

[0073] The product form of the embodiments of the present application can be a text generation application program. The text generation application program can run on a terminal device or a server on the cloud side.

[0074] In a possible implementation, the text generation application program can implement a text generation task based on input data (such as a question). The text generation application program can execute the text generation task in response to the input data (such as a question) to obtain a text generation result.

[0075] In a possible implementation, the user can open the text generation application program installed on the terminal device and input the input data (such as a question). The text generation application program can perform text generation on the input data by the method provided in the embodiments of the present application, and present the text generation result to the user (the presentation method can be but is not limited to display, save, upload to the cloud side, etc.).

[0076] In a possible implementation, a user can open a text generation application installed on a terminal device, and input input data. The text generation application can send the input data to a server at a cloud side. The server at the cloud side can perform text generation on the input data by using the method provided in the embodiments of the present application, and return the text generation result to the terminal device. The terminal device can present the text generation result to the user (the presentation manner can be, but is not limited to, display, save, upload to the cloud side, etc.).

[0077] Next, the text generation application in the embodiments of the present application is introduced from the functional architecture and the product architecture implementing the function respectively.

[0078] Referring to FIG. 1B, FIG. 1B is a schematic diagram of the functional architecture of the text generation application in the embodiments of the present application:

[0079] In a possible implementation, as shown in FIG. 1B, the text generation application 102 can receive input parameters 101 (for example, containing input data) and generate a text generation result 103. The text generation application 102 can be executed on, for example, at least one computer system, and includes computer code that, when executed by one or more computers, causes the computers to perform a natural language model trained by the method provided in the embodiments of the present application.

[0080] Referring to FIG. 2, FIG. 2 is a schematic diagram of the entity architecture of the text generation application in the embodiments of the present application:

[0081] Referring to FIG. 2, FIG. 2 shows a schematic diagram of a system architecture. The system can include a terminal 100 and a server 200. The server 200 can include one or more servers (one server is shown in FIG. 2 as an example), and the server 200 can provide a text generation service for one or more terminals.

[0082] The terminal 100 can install a text generation application or open a webpage related to text generation. The application and the webpage can provide an interface. The terminal 100 can receive parameters input by a user on the text generation interface, and send the parameters to the server 200. The server 200 can obtain a processing result based on the received parameters, and return the processing result to the terminal 100.

[0083] It should be understood that, in some optional implementations, the terminal 100 can also complete the action of obtaining a processing result based on received parameters by itself, without the cooperation of the server. The embodiments of the present application are not limited in this regard.

[0084] Next, the product form of the terminal 100 in FIG. 2 is described.

[0085] The terminal 100 in the embodiments of the present application can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), and the like, and the embodiments of the present application do not make any limitation in this regard.

[0086] FIG. 3 shows a schematic diagram of an optional hardware structure of the terminal 100.

[0087] Referring to FIG. 3, the terminal 100 can include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160 (optional), a speaker 161 (optional), a microphone 162 (optional), a processor 170, an external interface 180, a power supply 190, and the like. Those skilled in the art can understand that FIG. 3 is merely an example of the terminal or the multi-functional device, and does not constitute a limitation on the terminal or the multi-functional device, which can include more or fewer components than those shown, or combine some components, or different components.

[0088] The input unit 130 can be used to receive inputted digital or character information, and to generate key signal input related to user settings and function control of the portable multi-functional device. Specifically, the input unit 130 can include a touch screen 131 (optional) and / or other input devices 132. The touch screen 131 can collect touch operations of a user thereon or adjacent thereto (such as operations of the user using a finger, a joint, a stylus, or any suitable object on or adjacent to the touch screen), and drive corresponding connected devices according to a pre-set program. The touch screen can detect touch actions of the user on the touch screen, convert the touch actions into touch signals and send the touch signals to the processor 170, and can receive commands from the processor 170 and execute the commands; the touch signals at least include touch point coordinate information. The touch screen 131 can provide an input interface and an output interface between the terminal 100 and the user. In addition, the touch screen can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch screen 131, the input unit 130 can also include other input devices. Specifically, the other input devices 132 can include one or more of, but are not limited to, a physical keyboard, function keys (such as volume control keys, on-off keys, and the like), a trackball, a mouse, a joystick, and the like.

[0089] The other input devices 132 can receive inputted input data and the like.

[0090] The display unit 140 can be used to display information input by a user or information provided to the user, various menus of the terminal 100, an interactive interface, file display, and / or playing of any kind of multimedia file. In embodiments of the present application, the display unit 140 can be used to display an interface of a text generation application, a generated text generation result, and the like.

[0091] The memory 120 can be used to store instructions and data. The memory 120 can mainly include a storage instruction area and a storage data area. The storage data area can store various data, such as multimedia files, texts, and the like. The storage instruction area can store software units required by the operating system, applications, at least one function, and the like, or their subsets, extended sets. It can also include a non-volatile random access memory. It provides the processor 170 with software and applications that include managing hardware, software, and data resources in a computing processing device, supporting control software, and applications. It is also used for storage of multimedia files, and storage of running programs and applications.

[0092] The processor 170 is the control center of the terminal 100, and connects various parts of the entire terminal 100 through various interfaces and lines. It performs various functions of the terminal 100 and processes data by running or executing instructions stored in the memory 120 and calling data stored in the memory 120, thereby performing overall control of the terminal device. Optionally, the processor 170 can include one or more processing units. Preferably, the processor 170 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application program, and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 170. In some embodiments, the processor, the memory, and the like can be implemented on a single chip, and in some embodiments, they can also be implemented on separate chips. The processor 170 can also be used to generate corresponding operation control signals to the corresponding components of the computing processing device, read and process data in the software, especially read and process data and programs in the memory 120, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0093] The memory 120 can be used to store software codes related to the data processing method, and the processor 170 can execute the steps of the chip data processing method, or can schedule other units (such as the above-mentioned input unit 130 and display unit 140) to realize corresponding functions.

[0094] The RF unit 110 (optional) can be used for transmitting and receiving information or signals in the process of information or communication, for example, receiving the downlink information of the base station, and processing by the processor 170. In addition, the uplink data is sent to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the RF unit 110 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.

[0095] In the embodiments of the present application, the RF unit 110 can send input data to the server 200 and receive the text generation result sent by the server 200.

[0096] It should be understood that the RF unit 110 is optional, which can be replaced by other communication interfaces, for example, a network interface.

[0097] The terminal 100 also includes a power supply 190 (such as a battery) for supplying power to each component. Preferably, the power supply can be logically connected to the processor 170 through a power management system, so as to realize the functions of managing charging, discharging, and power consumption management through the power management system.

[0098] The terminal 100 also includes an external interface 180, which can be a standard Micro USB interface or a multi-pin connector, and can be used for connecting the terminal 100 and other devices for communication, or for connecting a charger to charge the terminal 100.

[0099] Although not shown, the terminal 100 can also include a flash, a wireless fidelity (WiFi) module, a Bluetooth module, different function sensors, etc., which will not be described here. Part or all of the methods described below can be applied in the terminal 100 as shown in FIG. 3.

[0100] Next, the product form of the server 200 in FIG. 2 is described;

[0101] FIG. 4 provides a structural schematic diagram of a server 200. As shown in FIG. 4, the server 200 includes a bus 201, a processor 202, a communication interface 203, and a memory 204. The processor 202, the memory 204, and the communication interface 203 communicate through the bus 201.

[0102] The bus 201 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in FIG. 4, but it does not mean that there is only one bus or only one type of bus.

[0103] The processor 202 can be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), etc.

[0104] The memory 204 can include a volatile memory, such as a random access memory (RAM). The memory 204 can also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard drive drive (HDD), or a solid state drive (SSD).

[0105] The memory 204 can be used to store software codes related to the data processing method, and the processor 202 can execute the steps of the chip data processing method or schedule other units to realize the corresponding functions.

[0106] It should be understood that the terminal 100 and the server 200 described above can be centralized or distributed devices, and the processors (for example, the processor 170 and the processor 202) in the terminal 100 and the server 200 can be hardware circuits (for example, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a general-purpose processor, a digital signal processor (DSP), a microprocessor, a microcontroller, or the like) or a combination of the hardware circuits. For example, the processor can be a hardware system with an instruction execution function, such as a CPU, a DSP, or the like, or a hardware system without an instruction execution function, such as an ASIC, an FPGA, or the like, or a combination of the hardware system without an instruction execution function and the hardware system with an instruction execution function.

[0107] It should be understood that the steps related to the model inference process in the embodiments of the present application involve AI-related operations. When performing AI operations, the instruction execution architecture of the terminal device and the server is not limited to the architecture of the processor combined with the memory described above. The system architecture provided by the embodiments of the present application will be described in detail below with reference to FIG. 5.

[0108] FIG. 5 is a schematic diagram of a system architecture provided by an embodiment of the present application. As shown in FIG. 5, the system architecture 500 includes an execution device 510, a training device 520, a database 530, a client device 540, a data storage system 550, and a data collection device 560.

[0109] The execution device 510 includes a computing module 511, an I / O interface 512, a preprocessing module 513, and a preprocessing module 514. The target model / rule 501 can be included in the computing module 511, and the preprocessing module 513 and the preprocessing module 514 are optional.

[0110] The execution device 510 can be a terminal device or a server running a text generation application.

[0111] The data collection device 560 is configured to collect training samples. The training samples can be program files (including program codes and program input data) and the like. After the training samples are collected, the data collection device 560 stores the training samples in the database 530.

[0112] The training device 520 can train a neural network to be trained based on the training samples maintained in the database 530 to obtain the target model / rule 501.

[0113] It should be noted that, in actual application, the training samples maintained in the database 530 do not necessarily all come from the collection of the data collection device 560, but can also be received from other devices. In addition, it should be noted that the training device 520 does not necessarily train the target model / rule 501 based on the training samples maintained in the database 530, but can also obtain training samples from the cloud or other places for model training. The above description should not be regarded as a limitation of the embodiments of the present application.

[0114] The target model / rule 501 trained by the training device 520 can be applied to different systems or devices, such as the execution device 510 shown in FIG. 5. The execution device 510 can be a terminal such as a mobile phone terminal, a tablet computer, a notebook computer, an augmented reality (AR) / virtual reality (VR) device, a vehicle-mounted terminal, etc., and can also be a server, etc.

[0115] Specifically, the training device 520 can deliver the trained model to the execution device 510.

[0116] In FIG. 5, the execution device 510 is configured with an input / output (I / O) interface 512 for data interaction with external devices. The user can input data (for example, input data in the embodiments of the present application) to the I / O interface 512 through the client device 540.

[0117] The pre-processing modules 513 and 514 are used for pre-processing the input data received by the I / O interface 512. It should be understood that there can be no pre-processing modules 513 and 514 or only one pre-processing module. When there is no pre-processing module 513 and 514, the input data can be directly processed by the calculation module 511.

[0118] During the pre-processing of the input data by the execution device 510 or during the calculation process performed by the calculation module 511 of the execution device 510, the execution device 510 can call the data, code, etc. in the data storage system 550 for corresponding processing, or store the data, instructions, etc. obtained by the corresponding processing in the data storage system 550.

[0119] Finally, the I / O interface 512 provides the processing result (for example, the text generation result, etc.) to the client device 540, thereby providing the user.

[0120] In the case shown in FIG. 5, the user can manually give input data, which can be operated through the interface provided by the I / O interface 512. In another case, the client device 540 can automatically send input data to the I / O interface 512, and if the user's authorization is required for the client device 540 to automatically send input data, the user can set the corresponding permission in the client device 540. The user can view the results output by the execution device 510 in the client device 540, and the specific presentation form can be a specific way such as display, sound, action, etc. The client device 540 can also act as a data collection end, collect the input data input into the I / O interface 512 and the output results output from the I / O interface 512 as new sample data, and store them into the database 530. Of course, the collection can also be performed without the client device 540, and the I / O interface 512 can directly store the input data input into the I / O interface 512 and the output results output from the I / O interface 512 as new sample data into the database 530.

[0121] It is worth noting that FIG. 5 is only a schematic diagram of a system architecture provided by an embodiment of the present application, and the positional relationship between the devices, components, modules and the like shown in the figure does not constitute any limitation. For example, in FIG. 5, the data storage system 550 is an external memory relative to the execution device 510, and in other cases, the data storage system 550 can also be placed in the execution device 510. It should be understood that the above execution device 510 can be deployed in the client device 540.

[0122] From the inference side of the model:

[0123] In the embodiment of the present application, the computing module 511 of the execution device 510 can obtain the code stored in the data storage system 550 to implement the steps related to the model inference process in the embodiment of the present application.

[0124] In the embodiment of the present application, the computing module 511 of the execution device 510 can include a hardware circuit (such as an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a general-purpose processor, a digital signal processor (DSP), a microprocessor or a microcontroller, etc.), or a combination of these hardware circuits. For example, the training device 520 can be a hardware system with an execution instruction function, such as a CPU, a DSP, etc., or a hardware system without an execution instruction function, such as an ASIC, an FPGA, etc., or a combination of the hardware system without an execution instruction function and the hardware system with an execution instruction function.

[0125] Specifically, the computing module 511 of the execution device 510 can be a hardware system with an execution instruction function, and the steps related to the model inference process provided in the embodiments of the present application can be software codes stored in the memory. The computing module 511 of the execution device 510 can obtain the software codes from the memory and execute the obtained software codes to implement the steps related to the model inference process provided in the embodiments of the present application.

[0126] It should be understood that the computing module 511 of the execution device 510 can be a combination of a hardware system without an execution instruction function and a hardware system with an execution instruction function, and part of the steps related to the model inference process provided in the embodiments of the present application can also be implemented by the hardware system without an execution instruction function in the computing module 511 of the execution device 510, which is not limited here.

[0127] From the training side of the model:

[0128] In the embodiments of the present application, the training device 520 can obtain the codes stored in the memory (not shown in FIG. 5, which can be integrated in the training device 520 or deployed separately from the training device 520) to implement the steps related to the model training in the embodiments of the present application.

[0129] In the embodiments of the present application, the training device 520 can include hardware circuits (such as application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), general-purpose processors, digital signal processors (DSPs), microprocessors or microcontrollers, etc.), or combinations of these hardware circuits. For example, the training device 520 can be a hardware system with an execution instruction function, such as a CPU, a DSP, etc., or a hardware system without an execution instruction function, such as an ASIC, an FPGA, etc., or a combination of the hardware system without an execution instruction function and the hardware system with an execution instruction function.

[0130] It should be understood that the training device 520 can be a combination of a hardware system without an execution instruction function and a hardware system with an execution instruction function, and part of the steps related to the model training provided in the embodiments of the present application can also be implemented by the hardware system without an execution instruction function in the training device 520, which is not limited here.

[0131] II. Text generation type cloud service provided by the server:

[0132] In a possible implementation, the server can provide the text generation service for the terminal side through an application programming interface (API).

[0133] The terminal device can send relevant parameters (for example, containing input data) to the server through the API provided by the cloud, the server can obtain a processing result (for example, a text generation result, etc.) based on the received parameters, and return the processing result to the terminal.

[0134] The description of the terminal and the server can be described in the above embodiments, which will not be repeated here.

[0135] As shown in FIG. 6, a flow of using a text generation cloud service provided by a cloud platform is shown.

[0136] 1. Open and purchase a text generation service.

[0137] 2. The user can download the software development kit (SDK) corresponding to the text generation service. Usually, the cloud platform provides multiple development versions of SDKs for users to select according to the needs of the development environment, such as JAVA version SDK, python version SDK, PHP version SDK, Android version SDK, etc.

[0138] 3. The user downloads the corresponding version of the SDK to the local according to the needs, imports the SDK project into the local development environment, and configures and debugs in the local development environment. The local development environment can also develop other functions, so as to form an application that integrates text generation capabilities.

[0139] 4. During the use of the text generation application, when text generation is needed, the API call of text generation can be triggered. When the application triggers text generation, an API request is initiated to the running instance of the text generation service in the cloud environment, wherein the input data is carried in the API request. The running instance in the cloud environment processes the input data to obtain a processing result (for example, a text generation result, etc.).

[0140] 5. The cloud environment returns the processing result to the application, thereby completing a text generation service call.

[0141] In addition, in addition to the application program and the cloud service, the implementation form of the present application can also be in the large model application SDK.

[0142] Since the present application embodiment involves the application of a large number of neural networks, in order to facilitate understanding, the related terms and concepts related to neural networks involved in the present application embodiment will be introduced first.

[0143] (1) Neural network

[0144] The neural network can be composed of neural units, which can refer to an operation unit taking xs (i.e., input data) and an intercept 1 as inputs, and the output of the operation unit can be:

[0145] where s = 1, 2, … n, n is a natural number greater than 1, Ws is the weight of xs, b is the bias of the neural unit. f is an activation function of the neural unit, which is used to introduce a nonlinear characteristic into the neural network to convert the input signal in the neural unit into an output signal. The output signal of the activation function can be used as the input of the next layer of the convolution layer, and the activation function can be a sigmoid function. The neural network is a network formed by connecting a plurality of the above single neural units, i.e., the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, and the local receptive field can be a region composed of a plurality of neural units.

[0146] (2) Back propagation algorithm

[0147] The convolutional neural network can use a back propagation (BP) algorithm to correct the size of the parameters in the initial super-resolution model during the training process, so that the reconstruction error loss of the super-resolution model becomes smaller and smaller. Specifically, the forward transmission of the input signal until the output will produce an error loss, and the initial super-resolution model parameters are updated by back propagating the error loss information, so as to make the error loss converge. The back propagation algorithm is a back propagation movement dominated by error loss, aiming to obtain the optimal parameters of the super-resolution model, such as the weight matrix.

[0148] (3) Loss function

[0149] In the process of training a deep neural network, because the output of the deep neural network is expected to be as close as possible to the value that is truly intended to be predicted, the weight vectors of each layer of the neural network can be updated according to the difference between the predicted value of the current network and the truly intended target value, after comparing the two. (Of course, before the first update, there is usually an initialization process, that is, the parameters of each layer of the deep neural network are pre-configured.) For example, if the predicted value of the network is too high, adjust the weight vector to make it predict lower, and keep adjusting until the deep neural network can predict the truly intended target value or a value very close to the truly intended target value. Therefore, it is necessary to define "how to compare the difference between the predicted value and the target value", which is the loss function or the objective function, which is an important equation for measuring the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, and then the training of the deep neural network becomes a process of trying to minimize this loss.

[0150] (4) Agent: an independent entity that can think and interact with the environment.

[0151] (5) Large language model-based agent: Based on large language models, combined with task planning, decision-making mechanisms and environmental interaction capabilities, enabling it to perform complex tasks in dynamic environments.

[0152] (6) Large language model-based multi-agent collaboration system: a system composed of multiple large language model-based agents. Agents in the system usually have different domain skills and professional knowledge, and work together to complete tasks through multiple interactions with the environment.

[0153] (7) Reward function: defines the goal of agent learning. At each step of interaction, the environment will pass a value to the agent, called reward. The goal of the agent is to maximize the sum of total rewards during the interaction process.

[0154] In recent years, with the emergence of large language models, large language model-based agents are becoming a research hotspot in the field of artificial intelligence. By utilizing the powerful language expression and reasoning planning capabilities of large models, large model-based autonomous agents can effectively understand and generate instructions, interact and make decisions in complex environments, and have achieved good results in various downstream tasks. To cope with more complex task scenarios, based on single-agent research, researchers propose to build multi-agent systems, which can split complex tasks into several sub-tasks and assign them to multiple agents with different domain skills and professional knowledge.

[0155] The division of labor of the agent in the multi-agent system mainly depends on the prior knowledge of the researchers, and the accuracy of the generated reply is poor, and the generation time is also longer. Therefore, how to better improve the generation accuracy of the reply through the multi-agent is a technical problem to be solved.

[0156] To improve the performance of multi-agent systems in various task scenarios, researchers have designed various multi-agent collaboration frameworks, and encoded the role division and cooperation mode of the agents in the form of prompts. However, due to the limitations of the context understanding ability of large models and the complexity of division and cooperation, such frameworks cannot fully utilize the collaboration ability of agents. How to better clarify the division of labor and communication and cooperation is a key step to improve the problem-solving ability of multi-agent systems. In this regard, a natural idea is to supervise and fine-tune the agent collaboration process. However, this will result in unacceptable computational resource consumption and affect the generalization ability of large language models. Inspired by the reflection and progress of human collaborators in team cooperation, the embodiments of the present application propose to optimize the multi-agent collaboration process with the help of an agent self-reflection mechanism. This self-reflection mechanism can convert numerical rewards from the environment into feedback in the form of text and add it as additional context to the prompt of the agent decision model. Such a structure can improve the role understanding and communication and cooperation ability of the agent in a specific task without losing the general ability of the agent.

[0157] Although the self-reflection mechanism can iteratively optimize the prompt of the decision model, effective reflection still requires the model to have sufficient understanding of complex scenarios to reasonably analyze the reasons for task failure, which poses a challenge to large language models with frozen parameters. In this regard, Retroformer proposes to use an open-source language model (such as Llama, ChatGLM) as a reflector and use numerical feedback from the environment as a supervision signal to fine-tune the reflector in a single-agent scenario. However, directly extending this method to the multi-agent collaboration scenario has many problems. On the one hand, numerical feedback from the environment can only evaluate the overall effectiveness of cooperation, and cannot score the contribution of individual agents in the system. Directly using it as a supervision signal in a multi-agent system will cause serious credit allocation problems, making it difficult to encourage agents in the system to improve individual behavior from the perspective of collective interests, or even make personal sacrifices for the collective interests. On the other hand, in a multi-agent system, the reflector is required to perceive the role information of the agent, understand the interaction relationship between the agents, and make targeted improvements according to the task division of the role. In this regard, a natural idea is to train a reflector for each agent. However, as the number of agents increases, the time and computational overhead required for training the reflector will also increase, which poses a challenge to the application in actual scenarios.

[0158] Therefore, for the multi-agent cooperation scenario, the embodiment of the present application proposes to train a plug-and-play reflector COPPER, which can efficiently reflect the multi-agent cooperation process, thereby improving the performance of the multi-agent system on complex tasks. To solve the first challenge, the embodiment of the present application proposes to construct counterfactual rewards as a supervision signal for individual agent fine-tuning. Specifically, the embodiment of the present application first adds the reflection of all agents to the prompt of the corresponding decision model to obtain the overall reward. Then, the reflection of each agent is removed in turn, and the process of interacting with the environment is repeated, and the reward obtained is used as the marginal reward. The difference between the overall reward and the marginal reward is used as the contribution evaluation of the individual agent in the cooperation process, and is used as a supervision signal for fine-tuning of the agent reflector. The reflector is optimized by the proximal policy optimization method, thereby maximizing the environment reward. To solve the second challenge, considering the homogeneity between different reflectors, i.e., their action space (reflection) is consistent, and their optimization goal is also completely consistent (auxiliary solution to the overall task), the embodiment of the present application proposes that all agents share the same reflector, and through the careful design of the prompt, the shared reflector can perceive the role information of different agents. This sharing mechanism not only reduces the demand for computing resources, but also makes the model training data more, and the training more stable.

[0159] The division of labor of the agent in the multi-agent system mainly depends on the prior knowledge of the researchers. How to better clarify the division of labor and communication and cooperation is a key step to improve the problem solving ability of the multi-agent system.

[0160] 2) Limited by the context understanding ability of large models and the complexity of division of labor and cooperation, such frameworks cannot fully exert the cooperation ability of agents.

[0161] For the multi-agent cooperation scenario, the application proposes to train a plug-and-play reflector COPPER to efficiently reflect the multi-agent cooperation process, thereby improving the performance of the multi-agent system on complex tasks. To solve the first challenge, the application proposes to construct counterfactual rewards as a supervision signal for individual agent fine-tuning. Specifically, the application first adds the reflection of all agents to the prompt of the corresponding decision model to obtain the overall reward. Then, the reflection of each agent is removed one by one, and the process of interacting with the environment is repeated, and the reward obtained is used as the edge reward. The difference between the overall reward and the edge reward is used as the contribution evaluation of the individual agent in the cooperation process, and is used as a supervision signal for fine-tuning of the agent reflector. The reflector is optimized through the proximal policy optimization method, thereby maximizing the environment reward. To solve the second challenge, considering the homogeneity between different reflectors, i.e., their action space (reflection) is consistent, and their optimization goal is also completely consistent (to assist in solving the overall task), the application proposes that all agents share the same reflector, and through a carefully designed prompt, the shared reflector can perceive the role information of different agents. This sharing mechanism not only reduces the demand for computing resources, but also makes the model training data more and the training more stable.

[0162] The data processing method of the present application will be introduced below. Referring to the flowchart of the data processing method shown in FIG. 7, the method can be executed by a data processing system. The method comprises:

[0163] 701, obtaining a question;

[0164] 702, obtaining a plurality of first interaction information through interaction between a plurality of agents according to the question, the plurality of first interaction information comprising a first reply to the question;

[0165] To cope with more complex task scenarios, complex tasks can be decomposed and assigned to multiple expert agents, and a multi-agent system is constructed to improve the performance of the model on complex tasks. For the multi-hop reasoning question answering scenario, the present application embodiment constructs a "teacher-student" cooperation mode, and in the mathematics and chess scenarios, the present application embodiment adopts a cooperative mechanism of cooperative debate. The agent cooperation mechanism adopted by the present application embodiment is shown in FIG. 8.

[0166] Specifically, the agents in the environment can be set to speak in a specified order, and efficient communication and cooperation between agents can be achieved by maintaining a shared message pool. For the kth question in the environment, at time t, agent i (i = t mod N) first takes out the previous historical interaction record from the message pool and obtains the current environment state s k,t , and the agent decision process can be represented as:

[0167] Where p i The role and action space of the current agent are defined. After making a decision, the agent will send the new message s. k,t ,a k,t Add it to the message pool. However, in actual interaction, due to the number of agents and decision steps, there may be a problem of excessively long historical interaction records. Therefore, for each agent in the environment, this embodiment introduces a context model to iteratively update the interaction history and uses it as the agent's short-term memory. The short-term memory update process can be represented as:

[0168] The corresponding agent's decision-making process can be represented as:

[0169] 703. When the first response is not the correct answer to the prompt, based on the plurality of first interaction information, the reflector determines the cause information corresponding to each of the agents, and the cause information is the reason for obtaining the incorrect response;

[0170] In one possible implementation, the same reflector is used when determining the cause information corresponding to each of the agents.

[0171] This application proposes a shared reflector for constructing a multi-agent collaborative system. Considering the homogeneity among different reflectors—that is, their action spaces (reflections) are consistent, and their optimization objectives are also completely identical (aiding in solving the overall task)—this application proposes that all agents share the same reflector, and through carefully designed prompts, the shared reflector can perceive the role information of different agents. This sharing mechanism not only reduces the demand for computational resources but also results in more training data and more stable training.

[0172] The method proposed in this application can reflect on the interaction information obtained by multiple agents when the collaboration among multiple agents fails to obtain a correct response. The purpose of the reflection is to determine the reason why the correct response was not obtained and the reasons of each agent. Based on this information, the agent participates in the next round of multi-agent collaboration to determine the response. This can improve the collaboration capability of the multi-agent system, the accuracy of the generated response, and reduce the time cost of the generation process.

[0173] Inspired by the progress of human collaborators in the process of team cooperation, the embodiments of the present application propose to optimize the multi-agent cooperation process with the help of the self-reflection mechanism of the agent. This self-reflection mechanism can convert the numerical rewards from the environment into feedback in the form of text and add it as additional context to the prompts of the agent decision model. Such a structure can improve the role understanding and communication and cooperation ability of the agent in a specific task without losing the general ability of the agent.

[0174] To improve the cooperation ability of the multi-agent system in a specific scenario while maintaining the generality of the agent decision model, the embodiments of the present application propose to introduce a multi-agent reflection mechanism. With the guidance of the environmental reward signal, the reflection generated by the reflector is used to continuously optimize the prompts of the decision model. FIG. 9A shows the details of the multi-agent reflection framework. Specifically, the embodiments of the present application add a reflector Reflector i to each agent i in the system. The reflector takes the interaction trajectory of the agent system and the environment, the reward signal of the environment as input, reflects the problems existing in the cooperation process of the agent, and formulates a new action plan. The role information p i of the current agent is also added to the input of the reflector, so as to obtain a reflection for the action space of the current agent. For the agent i in the environment, the reflection process of the reflector Reflector i can be defined as:

[0175] where k represents the kth problem in the environment, λ represents the λth attempt of the problem, and T represents the length of the interaction trajectory in the λth attempt. Since the short-term memory of the agent is updated iteratively, the short-term memory at time T contains the complete action information of the agent in this interaction. Taking the short-term memory of all agents in the system at time T as the input of the agent, the current agent obtains a global perspective and effectively reflects the problems existing in the cooperation process.

[0176] The reflection generated by the agent i is stored in its long-term memory. At the λth attempt of the problem k, the long-term memory of the agent i contains all the reflections generated by the λ-1th attempt, that is, The embodiments of the present application also add the long-term memory of the agent to the prompts of its decision model, thereby helping the agent to make better decisions. Therefore, for the λth attempt of the problem k, the decision process of the agent can be defined as:

[0177] By utilizing the agent's reflection mechanism and guided by environmental reward signals, the agent can learn from historical interaction records, continuously improve its cooperative ability, and realize the self-learning and self-evolution of the multi-agent collaborative system.

[0178] 704. Based on the question and the reason information corresponding to each of the intelligent agents, a second answer to the question is obtained through the interaction between the multiple intelligent agents.

[0179] In one possible implementation, the plurality of agents includes a target agent; based on the question and the cause information corresponding to each agent, second interaction information corresponding to each agent can be obtained through the interaction between the plurality of agents, and the plurality of second interaction information includes the second response.

[0180] In one possible implementation, the second interaction information corresponding to the target agent is obtained by: based on the question and the reason information corresponding to the target agent, the second interaction information of the target agent is obtained through the interaction between the target agent and other agents.

[0181] In one possible implementation, the second interaction information corresponding to the target agent is obtained in the following way: based on the question and the reason information corresponding to multiple agents including the target agent, the second interaction information of the target agent is obtained through the interaction between the target agent and other agents.

[0182] For example, for problem k in the environment, during the λth attempt, the intelligent agent system first interacts with the environment to generate an interaction trajectory τ. k,λ Subsequently, the environment generates a reward signal r. k,λ This application's embodiments convert environmental signals into natural language, guiding the intelligent agent to reflect on historical interactions. In this process, the Reflector... i Input The output is Will As training data, this embodiment uses the reflections generated by all agents in the system on K tasks as training data for a shared reflector. The training data D can be defined as follows:

[0183] Where Λ is the maximum number of attempts and K is the total number of problems in the environment.

[0184] In a possible implementation, the method further includes: obtaining a label of the cause information corresponding to each of the agents and a weight corresponding to the cause information; the weight indicates a degree of positive influence of using the reflector on accuracy of the obtained second interaction information compared with not using the reflector; and the reflector is fine-tuned according to the label and the weight.

[0185] Embodiments of the present application propose to construct counterfactual rewards as a supervision signal for fine-tuning of individual agents, and use a proximal policy optimization method to fine-tune the reflector of the agent. First, the reflections of all agents are added to the hints of the corresponding decision model to obtain the overall reward. Then, the reflection of each agent is removed in turn, and the process of interaction with the environment is repeated, and the reward obtained is used as the marginal reward. The difference between the overall reward and the marginal reward is used as the contribution evaluation of the individual agent in the cooperation process, and is used as a supervision signal for fine-tuning of the reflector of the agent. The reflector is optimized by the proximal policy optimization method, so as to maximize the environment reward.

[0186] In the fine-tuning process of the reflector, the reflections generated by the agents need to be scored, that is, the score of each piece of training data Score construction A natural idea is to use the difference between the scores of the two attempts as the score of the current reflection, that is, But this will result in the same score for the reflections of the agents in the environment, and cannot score the contribution of the agent reflections in the cooperation individually. Therefore, embodiments of the present application propose to introduce counterfactual scores to obtain individual evaluations for the agents in the environment. The construction process of the counterfactual reward is shown in FIG. 9B.

[0187] Specifically, embodiments of the present application use the difference between the scores of the two attempts as the overall reward of the agent system, and then removes the reflection of a certain agent i in the environment to obtain a new round of iteration score Such a reward is called a marginal reward The score of the reflection generated by the agent i is obtained by subtracting the overall reward from the marginal reward, that is, The obtained reward is called a counterfactual reward, and is used as the training data of the shared reflector.

[0188] Embodiments of the present application can include three stages of supervised training, reward model training, and proximal policy optimization.

[0189] In the first stage, the reflector is supervised training fine-tuning using positive samples in the collected reflection data, and the loss function of model training in this stage is as follows:

[0190] Where x represents the input of the reflector, and y represents the generated text.

[0191] In the second stage, considering the construction cost of the training data, unlike the standard RLHF, the embodiments of the present application do not construct positive and negative samples, but predict the reflection score by training a regression model. The embodiments of the present application use the mean square error loss function to optimize the reward model R φ Optimization. This process can be represented as:

[0192] In the third stage, the supervised trained model is further fine-tuned using the near segment strategy optimization method, and the trained reward model is used to score the output of the model. The training goal of this process is to maximize the environmental reward, and its loss function is as follows:

[0193] The scenario applied by the embodiments of the present application is a multi-agent collaboration system based on a large language model. The main structure of this scenario is shown in FIG. 9C, which includes a multi-agent system, tools, an environment, and a memory.

[0194] The basic operation logic of the multi-agent system based on a large language model is as follows: first, the agents in the multi-agent system use the powerful natural language processing capabilities of the large language model to understand and process inputs, plan and reason, and generate preliminary task execution plans. Then, during task execution, agents rely on various tools to complete specific operations, such as generating text answers, executing code, or performing database queries, and dynamically interact with the external environment. Agents obtain feedback through interaction with the environment and store these feedback in the memory module. Next, the agents use the large language model to reflect on the stored feedback, analyze and summarize the problems and shortcomings in the execution process, and thus propose improvement strategies. The memory module not only records past experiences and feedback, but also supports the agents in the environment to update their knowledge base and decision-making model, supporting continuous self-learning and adaptation. In this way, the multi-agent system can continuously improve its task execution capabilities and environmental adaptability in the cycle of understanding, execution, storing feedback, reflection, and learning.

[0195] The core device of the embodiment of the present application is mainly a shared reflector fine-tuning framework of a multi-agent system. In the framework of the present application, the efficiency of the reflector fine-tuning of the multi-agent system is reduced while the personalized reflection ability is retained. The present application proposes to construct a shared reflection framework, and add agent role description to the reflection prompt. The multi-agent system obtains a series of task trajectories through continuous interaction with the environment. For the trajectories that answer incorrectly, the system uses the shared reflector to generate reflections for each agent, which will be added to the prompt in the next task answer of the agent. The reward score of the agent reflection in the environment is obtained through the counterfactual reward method. The counterfactual reward dataset is collected, and the RLHF algorithm is used to fine-tune the shared reflector.

[0196] The framework proposed in the embodiment of the present application is divided into two stages, stage one for counterfactual data collection, and stage two for shared reflector fine-tuning. In stage one, the embodiment of the present application first adds the reflections of all agents to the prompt of the corresponding decision model to obtain the overall reward. Then, the reflection of each agent is removed one by one, and the interaction process with the environment is repeated, and the reward obtained is used as the marginal reward. The difference between the overall reward and the marginal reward is used as the evaluation of the contribution of a single agent in the cooperation process, and is used as a supervision signal to fine-tune the agent reflector. In stage two, the present application uses the proximal policy optimization method to optimize the reflector, and maximizes the environmental reward to improve the reflection ability of the shared reflector of the multi-agent system.

[0197] For example, an algorithm process example corresponding to the present application is as follows:

[0198] Suppose the task is a multi-hop question answering reasoning task. For this task, we assume that there are two agents in the multi-agent system: a student agent and a teacher agent. Among them, the student agent can call a search engine to search for information related to the keyword, or select an answer to the question. The teacher agent can guide the subsequent actions of the student agent based on the problem information and the answer process of the student agent, including [Rethink] and [Continue] two options. The student agent and the teacher agent in the environment participate in question answering in turn until the student agent gives an answer or reaches the preset step limit.

[0199] Suppose the training set has 10 tasks, and the multi-agent system will answer the question at most Λ times. If the answer is correct, stop answering. If the answer is incorrect, generate a reflection, for example, "I got stuck in a loop when searching for a certain keyword last time, I will try to change the keyword next time". The newly generated reflection will be added to the prompt of the next round of agent to help the agent answer the question.

[0200] In the multi-hop question answering task, we use F1-Score (a value between 0 and 1) to evaluate the answer. Suppose that the multi-agent system scores 0.5 when answering the question for the first time. When answering the question for the first time, the multi-agent system scores 0.9. By subtracting the scores of the two rounds, we can obtain the overall reward of the multi-agent system (the increase brought by the student agent and the teacher agent) is 0.4. Then we delete the reflections of the student agent and the teacher agent when answering the question for the first time in turn, and obtain the scores of the new agent system for the first time are 0.7 and 0.8 respectively. It can be calculated that the marginal reward of the student agent and the teacher agent is 0.7-0.5=0.2 (excluding the task improvement brought by the reflection of the student agent), 0.8-0.5=0.3 (excluding the improvement brought by the reflection of the teacher agent). The reward of the student agent reflection is 0.4-0.2=0.2, and the reward of the teacher agent reflection is 0.4-0.3=0.1.

[0201] The reward of the single agent reflection is obtained according to the described counterfactual reward construction method. These data are collected as the training data of the shared reflector.

[0202] The training of the shared reflector can be divided into three steps: first, the positive examples in the training set are selected to supervise the fine-tuning of the shared reflection; then a linear regression model is used to train the reward function; finally, the trained reward function is used to further fine-tune the shared reflector through the PPO algorithm.

[0203] Next, the technical solutions of the embodiments of the application are introduced by taking a mobile phone intelligent assistant as an example. In the intelligent assistant, the user can have a conversation with the agent system, and the assistant can answer the user's question based on existing knowledge or call tools to retrieve related data. In order to provide better user service, we assume that the intelligent assistant system is composed of multiple cooperative expert agents, such as music agents, encyclopedia agents, etc. For a user request, the agents discuss and obtain the final answer through cooperation.

[0204] For each agent in the mobile assistant agent system, a large model with fixed parameters and good general task performance is selected as the decision-making model, and a shared reflection model is set for the agent system. Through continuous reflection, the quality of user answers is continuously improved. The method of the embodiment of the application can be used to fine-tune the shared reflector in the system. First, the mobile assistant agent system is put online and interacts with the user, collects user feedback through certain strategies, and collects counterfactual establishment data in multiple feedbacks with the user, and then fine-tunes the shared reflector using the RLHF algorithm. The framework is a basic framework of a multi-agent collaboration system based on a large language model, has wide adaptability, is very flexible, and is easy to apply to various multi-agent collaboration scenarios.

[0205] The embodiments of the application select HotPotQA, GSM8K, and Checkmate in One Move to test the ability of the model in multi-hop reasoning question answering, mathematics, and chess. In the three scenarios, the embodiments of the application all use full matching as the model evaluation index. The following is an introduction to the data sets.

[0206] HotPotQA is a question and answer data set focusing on multi-hop reasoning based on facts, aiming to improve the explainability of question and answer systems. In this data set, the agent needs to reason between two or more Wikipedia paragraphs to obtain the answer. The data set contains a total of 90447 pairs of question and answer data.

[0207] The GSM8K data set is a collection of 8.5K math problems designed for elementary school students. These math problems are carefully crafted by human experts to ensure language diversity; mainly focusing on basic arithmetic operations such as addition, subtraction, multiplication, and division; each problem requires 2 to 8 steps of reasoning to obtain a solution.

[0208] The Checkmate in One Move data set is taken from the BIG-Bench Chess-State Tracking benchmark, aiming to evaluate the effectiveness of chess steps in chess. Given a sequence of moves that can checkmate in one move, the agent is required to predict the position of the move that can achieve the checkmate situation. The data set contains a total of 3500 chess games.

[0209] The embodiment of the application selects gpt-3.5-turbo as the decision model of the intelligent agent, and longchat-7b-16k as the reflection device to be fine-tuned. On the HotPotQA and Checkmate in One Move data sets, 2000 data are randomly selected as the training set, and 100 data are randomly selected as the test set. On the GSM8K data set, due to the high success rate of the intelligent agent, the number of reflections is small, and 3000 data are randomly selected as the training set, and 100 data are randomly selected as the test set. The reflection data of the reflection device fine-tuning are collected on the training set. Specifically, the maximum number of attempts is set to 5, the decision model temperature is 0, and the reflection model temperature is 0.9. In the test stage, in order to ensure the reproducibility of the results, the decision model temperature and the reflection device temperature are both set to 0. LoRA is used for efficient fine-tuning of the reflection device model, and the trl package of Hugging Face is used to realize each stage of RLHF. In the supervised training process and the near-end strategy optimization process, the training round number is searched in the range of {1, 2, 3, 4}, the batch size is searched in the range of {64, 128, 256}, and the learning rate is searched in the range of {1e-4, 2e-4, 3e-4, 5e-4}. Gpt-2 is selected as the reward model, and the learning rate is set to 5e-5, the training round number is set to 3, and the batch size is set to 16. The model training is performed on a 4-card A800 80G.

[0210] Figure 9D shows the experimental results of different methods after 5 attempts, where the deepest line represents COPPER, the lightest line represents Reflexion using GPT-3.5 as the reflection device, and the line of intermediate depth represents Reflexion using LongChat without fine-tuning as the reflection device. The gray line represents ReAct or CoT.

[0211] The experimental results show that the results of different methods on the three data sets show roughly the same pattern: (1) The self-reflection mechanism can make the model performance increase with the number of attempts, which shows that reflection can help the model summarize experience from historical errors, so as to avoid errors in the next round of attempts and improve task performance. (2) The method COPPER proposed in the embodiment of the application has a significant improvement in reflection performance compared to LongChat without fine-tuning as the reflection device, and even exceeds the method using GPT-3.5 as the reflection device. The possible reason is that the reflection device learns to reflect the reasons for the failure of the task based on the historical attempts in the fine-tuning process, and can deeply understand the role of the current intelligent agent in cooperation, so as to give targeted improvement schemes for the current intelligent agent in cooperation.

[0212] Compared with the prior art, the numerical reward from the environment is used as a supervision signal to fine-tune the reflector. On the one hand, the application embodiment proposes counterfactual rewards to evaluate the contribution of individual agents in the system, which to some extent alleviates the credit assignment problem. On the other hand, the application embodiment proposes a shared reflector mechanism, which enables the reflector to reflect on the role while reducing the computational resource demand in model training, improving the stability of training. Experimental results on three public data sets show that COPPER has stronger reflection ability compared with the baseline model. Compared with the initial success rate, COPPER improves by 31.8%, 18.5% and 86.4% on the HotPotQA, GSM8K and Checkmate in One Move data sets, respectively.

[0213] Referring to FIG. 10, FIG. 10 is a structural schematic of a data processing apparatus provided by an embodiment of the application. As shown in FIG. 10, a data processing apparatus 1000 provided by an embodiment of the application comprises:

[0214] The question acquisition module 1001 is configured to acquire a question.

[0215] The reply determination module 1002 is configured to: obtain a plurality of first interaction information through interaction among a plurality of agents according to the question, the plurality of first interaction information comprising a first reply to the question; when the first reply is not a correct answer to the prompt, determine, through a reflector, reason information corresponding to each of the agents according to the plurality of first interaction information, the reason information being a reason for obtaining an incorrect reply; and obtain a second reply to the question through interaction among the plurality of agents according to the question and the reason information corresponding to each of the agents.

[0216] The second interaction information corresponding to the target agent is obtained in the following manner: the plurality of agents comprise a target agent.

[0217] The second interaction information corresponding to the target agent is obtained in the following manner: the plurality of agents comprise a target agent.

[0218] The reply determination module is specifically configured to:

[0219] The reply determination module is specifically configured to:

[0220] The second interaction information corresponding to the target agent is obtained in the following manner: the plurality of agents comprise a target agent.

[0221] According to the question and the cause information corresponding to the target agent, second interaction information of the target agent is obtained through interaction of the target agent and other agents.

[0222] The second interaction information corresponding to the target agent is obtained in the following manner:

[0223] According to the question and the cause information corresponding to the target agent, second interaction information of the target agent is obtained through interaction of the target agent and other agents.

[0224] The second interaction information corresponding to the target agent is obtained in the following manner:

[0225] The model training module is configured to obtain labels of the cause information corresponding to each agent and weights corresponding to the cause information, wherein the weights indicate a degree of positive influence of using the reflector on accuracy of the obtained second interaction information compared with not using the reflector.

[0226] The reflector is fine-tuned according to the labels and the weights.

[0227] In a possible implementation, the agent is a large language model (LLM).

[0228] Next, an execution device provided by an embodiment of the present application is introduced. Referring to FIG. 11, FIG. 11 is a structural schematic diagram of an execution device provided by an embodiment of the present application. Specifically, the execution device 1100 includes a receiver 1101, a transmitter 1102, a processor 1103, and a memory 1104 (wherein the number of processors 1103 in the execution device 1100 can be one or more, and one processor is taken as an example in FIG. 11). The processor 1103 can include an application processor 11031 and a communication processor 11032. In some embodiments of the present application, the receiver 1101, the transmitter 1102, the processor 1103, and the memory 1104 can be connected through a bus or other means.

[0229] The memory 1104 can include read-only memory and random access memory, and provide instructions and data to the processor 1103. A portion of the memory 1104 can also include non-volatile random access memory (NVRAM). The memory 1104 stores processor and operating instructions, executable modules or data structures, or subsets thereof, or expanded sets thereof, wherein the operating instructions can include various operating instructions for implementing various operations.

[0230] The processor 1103 controls the operation of the execution device. In a specific application, various components of the execution device are coupled together through a bus system, which can include a data bus, a power bus, a control bus, and a state signal bus, etc. However, for the sake of clarity, all the buses are referred to as a bus system in the figure.

[0231] The method disclosed in the above embodiments of the present application can be applied in the processor 1103 or implemented by the processor 1103. The processor 1103 can be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above method can be completed by integrated logic circuits or instruction forms of software in the processor 1103. The processor 1103 described above can be a general processor, a digital signal processor (DSP), a microprocessor or a microcontroller, and can further include an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. The processor 1103 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium in the art. The storage medium is located in the memory 1104, and the processor 1103 reads the information in the memory 1104, and combines the hardware to complete the steps of the method involving model inference process.

[0232] The receiver 1101 can be configured to receive inputted digital or character information, and to generate signal input related to relevant settings of the execution device and function control. The transmitter 1102 can be configured to output digital or character information through the first interface; the transmitter 1102 can also be configured to send instructions to the disk group through the first interface to modify data in the disk group; and the transmitter 1102 can further include a display device such as a display screen.

[0233] The embodiment of the present application further provides a server device. Please refer to FIG. 12, which is a schematic diagram of a structure of a server according to the embodiment of the present application. Specifically, the server 1200 is implemented by one or more servers. The server 1200 can be quite different due to different configurations or performances. The server 1200 can include one or more central processing units (CPUs) 1212 (for example, one or more processors) and a memory 1232, and one or more storage media 1230 (for example, one or more mass storage devices) for storing application programs 1242 or data 1244. The memory 1232 and the storage media 1230 can be temporary storage or persistent storage. The programs stored in the storage media 1230 can include one or more modules (not shown in the figure), and each module can include a series of instruction operations in the server. Furthermore, the central processing unit 1212 can be configured to communicate with the storage media 1230 and execute the series of instruction operations in the storage media 1230 on the server 1200.

[0234] The server 1200 can further include one or more power supplies 1226, one or more wired or wireless network interfaces 1250, and one or more input / output interfaces 1258; or one or more operating systems 1241, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0235] In the embodiment of the present application, the central processing unit 1212 is configured to execute the data processing method in the above embodiment.

[0236] The embodiment of the present application further provides a computer program product, which, when running on a computer, causes the computer to execute the steps performed by the execution device as described above, or causes the computer to execute the steps performed by the training device as described above.

[0237] The embodiment of the present application further provides a computer readable storage medium, which stores a program for signal processing. When the program runs on a computer, the computer is caused to execute the steps performed by the execution device as described above, or the computer is caused to execute the steps performed by the training device as described above.

[0238] The execution device, the training device or the terminal device provided by the embodiments of the present application can be a chip, which includes a processing unit, for example, a processor, and a communication unit, for example, an input / output interface, a pin or a circuit, etc. The processing unit can execute computer execution instructions stored in a storage unit, so that the chip in the execution device executes the data processing method described in the above embodiments, or so that the chip in the training device executes the data processing method described in the above embodiments. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit can also be a storage unit outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0239] Specifically, please refer to FIG. 13, which is a structural schematic diagram of a chip provided by the embodiments of the present application. The chip can be a neural network processor NPU 1300, which is mounted on a host CPU (Host CPU) as a coprocessor and is assigned tasks by the Host CPU. The core part of the NPU is an operation circuit 1303, which extracts matrix data in a memory and performs multiplication operation under the control of a controller 1304.

[0240] In some implementations, the operation circuit 1303 internally includes a plurality of processing units (PEs). In some implementations, the operation circuit 1303 is a two-dimensional systolic array. The operation circuit 1303 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 1303 is a general-purpose matrix processor.

[0241] For example, it is assumed that there are an input matrix A, a weight matrix B and an output matrix C. The operation circuit takes corresponding data of the matrix B from the weight memory 1302 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 1301 and performs matrix operation with the matrix B, and the partial result or final result of the obtained matrix is saved in an accumulator 1308.

[0242] The unified memory 1306 is used to store input data and output data. The weight data is transferred into the weight memory 1302 through the Direct Memory Access Controller (DMAC) 1305. The input data is also transferred into the unified memory 1306 through the DMAC.

[0243] The BIU is the Bus Interface Unit 1310, which is used for the interaction between the AXI bus and the DMAC and the instruction fetch buffer (IFB) 1309.

[0244] The BIU 1310 is used for the instruction fetch buffer 1309 to fetch instructions from the external memory, and is also used for the Direct Memory Access Controller 1305 to fetch the original data of the input matrix A or the weight matrix B from the external memory.

[0245] The DMAC is mainly used to transfer the input data in the external memory DDR into the unified memory 1306, or to transfer the weight data into the weight memory 1302, or to transfer the input data into the input memory 1301.

[0246] The vector calculation unit 1307 includes a plurality of operation processing units, which further process the output of the operation circuit 1303 as needed, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / full connection layer network calculation in neural networks, such as Batch Normalization, pixel-level summation, upsampling of feature planes, etc.

[0247] In some implementations, the vector calculation unit 1307 can store the processed output vector to the unified memory 1306. For example, the vector calculation unit 1307 can apply a linear function; or, a non-linear function to the output of the operation circuit 1303, such as linear interpolation on the feature planes extracted by the convolutional layer, and further, a vector of accumulated values to generate activation values. In some implementations, the vector calculation unit 1307 generates normalized values, pixel-level summation values, or both. In some implementations, the processed output vector can be used as activation input to the operation circuit 1303, such as for use in subsequent layers in the neural network.

[0248] The controller 1304 is connected to the instruction fetch buffer 1309, which is used to store instructions used by the controller 1304;

[0249] The unified memory 1306, the input memory 1301, the weight memory 1302, and the instruction memory 1309 are on-chip memories. The external memory is private to the NPU hardware architecture.

[0250] Any of the processors mentioned above can be a general central processing unit, a microprocessor, an ASIC, or one or more integrated circuits configured to control the execution of the programs mentioned above.

[0251] It should be noted that the apparatus embodiments described above are merely illustrative, and the units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., they can be located in one place or distributed on multiple network units. Some or all of the modules can be selected to achieve the purpose of the embodiment according to actual needs. In addition, the connection relationship between the modules in the apparatus embodiment provided in the present application indicates that there is a communication connection between them, which can be implemented as one or more communication buses or signal lines.

[0252] From the above description of the embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software and the necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. Generally, functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure for implementing the same function can also be various, such as analog circuits, digital circuits, or special circuits. However, for the present application, software program implementation is a better embodiment. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., including a plurality of instructions for making a computer device (which can be a personal computer, a training device, or a network device, etc.) execute the methods described in various embodiments of the present application.

[0253] In the above embodiments, all or part can be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, it can be implemented in the form of a computer program product in whole or in part.

[0254] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website, computer, training device or data center to another website, computer, training device or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be stored by the computer or a data storage device such as a training device, a data center, etc. integrated with one or more available media sets. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

Claims

1. A data processing method, characterized by, The method comprises: obtaining a question; obtaining a plurality of first interaction information through interaction between a plurality of agents according to the question, the plurality of first interaction information comprising a first reply to the question; when the first reply is not a correct answer to the prompt, determining, through a reflector, reason information corresponding to each of the agents according to the plurality of first interaction information, the reason information being a reason for obtaining an incorrect reply; obtaining a second reply to the question through interaction between the plurality of agents according to the question and the reason information corresponding to each of the agents.

2. The method of claim 1, wherein, The same reflector is used when determining the reason information corresponding to each of the agents.

3. The method according to claim 1 or 2, characterized in that, The plurality of agents comprises a target agent; The reply determination module is specifically configured to: obtain a plurality of second interaction information corresponding to each of the agents through interaction between the plurality of agents according to the question and the reason information corresponding to each of the agents, the plurality of second interaction information comprising the second reply.

4. The method of claim 3, wherein, The second interaction information corresponding to the target agent is obtained by: obtaining second interaction information of the target agent through interaction between the target agent and other agents according to the question and the reason information corresponding to the target agent.

5. The method of claim 3, wherein, The second interaction information corresponding to the target agent is obtained by: obtaining second interaction information of the target agent through interaction between the target agent and other agents according to the question and the reason information corresponding to the target agent.

6. The method according to any one of claims 1 to 5, characterized in that, The method further comprises: obtaining a label of the reason information corresponding to each of the agents and a weight corresponding to the reason information; the weight indicating a degree of positive influence of using the reflector on accuracy of the obtained second interaction information compared with not using the reflector; fine-tuning the reflector according to the label and the weight.

7. The method according to any one of claims 1 to 6, characterized in that, The agent is a large language model (LLM).

8. A data processing apparatus, characterized by, The device comprises: a question obtaining module configured to obtain a question; a reply determination module configured to obtain a plurality of first interaction information through interaction between a plurality of agents according to the question, the plurality of first interaction information comprising a first reply to the question; when the first reply is not a correct answer to the prompt, determine, through a reflector, reason information corresponding to each of the agents according to the plurality of first interaction information, the reason information being a reason for obtaining an incorrect reply; and obtain a second reply to the question through interaction between the plurality of agents according to the question and the reason information corresponding to each of the agents.

9. The apparatus of claim 8, wherein, The same reflector is used when determining the reason information corresponding to each of the agents.

10. The apparatus of claim 8 or 9, wherein, The plurality of agents comprises a target agent; The reply determination module is specifically configured to: According to the question and the cause information corresponding to each of the agents, second interaction information corresponding to each of the agents is obtained through interaction among the agents, and the second interaction information includes the second answer.

11. The apparatus of claim 10, wherein, The second interaction information corresponding to the target agent is obtained in the following manner: According to the question and the cause information corresponding to the target agent, second interaction information of the target agent is obtained through interaction between the target agent and other agents.

12. The apparatus of claim 10, wherein, The second interaction information corresponding to the target agent is obtained in the following manner: According to the question and the cause information corresponding to the target agent, second interaction information of the target agent is obtained through interaction between the target agent and other agents.

13. The apparatus of any one of claims 8 to 12, wherein, The apparatus further includes: A model training module configured to obtain labels of the cause information corresponding to each of the agents and weights corresponding to the cause information, wherein the weights indicate a degree of positive influence of using the reflector on accuracy of the obtained second interaction information compared to not using the reflector. The reflector is fine-tuned according to the labels and the weights.

14. The apparatus of any one of claims 8 to 13, wherein, The agents are large language models (LLMs).

15. A computing device, comprising: The computing device includes a memory and a processor, the memory stores code, and the processor is configured to execute the code, when the code is executed, the computing device performs the method of any one of claims 1 to 7.

16. A chip, characterized by The computing device includes at least one processing unit and interface circuitry, the interface circuitry is configured to provide program instructions or data for the at least one processing unit, and the at least one processing unit is configured to execute the program instructions to implement the method of any one of claims 1 to 7.

17. A cluster of computing devices, characterized in that, The computing device cluster includes at least one computing device, the at least one computing device includes at least one processor and at least one memory, and the at least one memory stores computer readable instructions; the at least one processor executes the computer readable instructions to enable the computing device cluster to perform the method of any one of claims 1 to 7.

18. A computer-readable storage medium, characterized in that, The computer readable instructions are configured to implement the method of any one of claims 1 to 7.

19. A computer program product, characterised in that, The computer readable instructions are configured to implement the method of any one of claims 1 to 7.

Images