Training method, device, electronic device and storage medium for multi-agent model

Abstract

The present application provides a training method and device for a multi-agent model; the method includes: a parameter aggregation device receives a first number of parameter value groups sent by each participating device, the parameter value group includes a parameter value of at least one unpredictable parameter; the parameter value groups sent by each participating device are summarized, and a second number of parameter value groups obtained by the summary are sent to the participating device; a second number of loss values ​​returned by each participating device are received, and the loss value is determined by the participating device based on the received parameter value group and the loss function of the multi-agent model; based on the second number of loss values, a first number of parameter value groups are selected from the second number of parameter value groups as the target parameter value group; the target parameter value group is sent to each participating device, so that the participating device updates the model parameters of the multi-agent model based on the target parameter value group. Through the present application, it is possible to quickly find suitable parameter values, making the update process of the multi-intelligence model more stable.

Description

Technical Field

[0001] The present application relates to artificial intelligence technology, and in particular to a training method, device, electronic device and computer-readable storage medium for a multi-agent model. Background Art

[0002] Federated learning technology is an emerging privacy protection technology that can effectively combine data from all parties for model training without leaving the local data.

[0003] The goal of horizontal federated learning is to use the limited data of all parties to jointly train a global model while ensuring data security. Because the global model uses the data of each participant for training, its model effect can be close to the situation where the data of each participant is trained together, and is significantly better than the effect of the model obtained by each participant based only on its own data. However, the use of multi-agent models is very different from traditional machine learning, and how to apply federated learning to solve the verification / training of multi-agent models is an unresolved problem.

[0004] In the related art, the method of updating the multi-agent model based on the federated averaging algorithm is to update the model parameters together with the horizontal federated learning framework. For example, the optimal parameters of each are regularly transmitted to the parameter aggregation device, and then the parameter aggregation device sends the weighted average parameter value to each participating device. However, the above method directly averages the parameters, which makes it difficult to reflect the real effect of each parameter point on the overall multi-agent model. Summary of the invention

[0005] The embodiments of the present application provide a training method, device, electronic device, computer-readable storage medium and computer program product for a multi-agent model, which can quickly find suitable parameter values, making the update process of the multi-agent model more stable and controllable.

[0006] The technical solution of the embodiment of the present application is implemented as follows:

[0007] The embodiment of the present application provides a training method for a multi-agent model, based on a federated learning system, wherein the system includes a parameter aggregation device and at least two participant devices, including:

[0008] The parameter aggregation device receives a first number of parameter value groups sent by each of the participant devices, wherein the parameter value groups include a parameter value of at least one unpredictable parameter;

[0009] Summarizing the parameter value groups sent by each of the participant devices, and sending a second number of parameter value groups obtained by summarizing to each of the participant devices respectively;

[0010] Receiving a second number of loss values ​​returned by each of the participant devices, wherein the loss values ​​are determined by the participant devices based on the received parameter value group and the loss function of the multi-agent model;

[0011] Based on each of the second number of loss values, selecting a first number of parameter value groups from each of the second number of parameter value groups as a target parameter value group;

[0012] The target parameter value group is sent to each of the participant devices respectively, so that each of the participant devices updates the model parameters of the multi-agent model based on the target parameter value group.

[0013] The present application embodiment provides a training device for a multi-agent model, comprising:

[0014] A first receiving module, configured to receive a first number of parameter value groups sent by each of the participant devices, wherein the parameter value groups include parameter values ​​of at least two unpredictable parameters;

[0015] A summarizing module, used for summarizing the parameter value groups sent by each of the participant devices, and sending the second number of parameter value groups obtained by summarizing to each of the participant devices respectively;

[0016] A second receiving module, configured to receive a second number of loss values ​​returned by each of the participant devices, wherein the loss values ​​are determined by the participant device based on the received parameter value group and the loss function of the multi-agent model;

[0017] a selection module, configured to select a first number of parameter value groups from the second number of parameter value groups as target parameter value groups based on the second number of loss values;

[0018] A sending module is used to send the target parameter value group to each of the participating devices respectively, so that each of the participating devices updates the model parameters of the multi-agent model based on the target parameter value group.

[0019] In the above solution, the selection module is further used to determine a third number of loss values ​​corresponding to each parameter value group in the second number of parameter value groups based on the second number of loss values ​​returned by each of the participant devices, wherein the third number is the number of the participant devices;

[0020] Determine a reference loss value corresponding to each of the parameter value groups based on a third number of loss values ​​corresponding to each of the parameter value groups;

[0021] The second number of parameter value groups are sorted in ascending order according to the reference loss values, and selection is started from the first parameter value group according to the sorting result until the first number of parameter value groups are selected as the target parameter value groups.

[0022] In the above solution, the selection module is further used to perform the following processing on each parameter value group in the second number of parameter value groups:

[0023] Summing a third number of loss values ​​corresponding to the parameter value group to obtain a summation result;

[0024] The summation result is used as the reference loss value corresponding to the parameter value group.

[0025] In the above solution, the selection module is further used to perform the following processing on each parameter value group in the second number of parameter value groups:

[0026] Averaging a third number of loss values ​​corresponding to the parameter value group to obtain a loss mean;

[0027] The loss mean is used as the reference loss value corresponding to the parameter value group.

[0028] In the above solution, the selection module is further used to sort the second number of loss values ​​returned by each of the participant devices respectively to obtain a loss value sequence corresponding to each of the participant devices;

[0029] Respectively obtain the order of the loss values ​​corresponding to each parameter value group in each of the loss value sequences;

[0030] Based on the order of each of the parameter value groups in each of the loss value sequences, a first number of parameter value groups are selected from the second number of parameter value groups as target parameter value groups.

[0031] In the above solution, the selection module is further used to, for each parameter value group, respectively average the ranking of the parameter value group in each loss value sequence to obtain a ranking average value;

[0032] A first number of parameter value groups are selected from the second number of parameter value groups as target parameter value groups according to a ranked average value of each of the parameter value groups in each of the loss value sequences.

[0033] In the above solution, the selection module is further used to, for each parameter value group, respectively sum the ranking of the parameter value group in each loss value sequence to obtain the sum of the rankings;

[0034] A first number of parameter value groups are selected from the second number of parameter value groups as target parameter value groups according to the sum of the order of the parameter value groups in the loss value sequences.

[0035] In the above solution, the device further includes an application module, and the application module is used to obtain predictable parameters and unpredictable parameters corresponding to the updated multi-agent model;

[0036] Initializing the predictable parameters and the unpredictable parameters to obtain a fourth number of initial parameter value groups;

[0037] Inputting the fourth number of initial parameter value groups into the multi-agent model to obtain a prediction result;

[0038] Based on the prediction result, the unpredictable parameter is updated to obtain a target value corresponding to the unpredictable parameter.

[0039] In the above solution, the application module is further used to initialize the fourth number of values ​​corresponding to the population distribution and the fourth number of values ​​corresponding to the propagation coefficient to obtain a fourth number of initial parameter value groups;

[0040] Inputting the fourth number of initial parameter value groups into the updated multi-agent model to obtain the number of confirmed cases and the number of deaths corresponding to the target disease;

[0041] Based on the number of confirmed cases and the number of deaths, the transmission coefficient is updated to obtain the transmission coefficient corresponding to the target disease.

[0042] An embodiment of the present application provides an electronic device, including:

[0043] A memory for storing executable instructions;

[0044] The processor is used to implement the training method of the multi-agent model provided in the embodiment of the present application when executing the executable instructions stored in the memory.

[0045] An embodiment of the present application provides a computer-readable storage medium storing executable instructions for causing a processor to execute and implement the training method of the multi-agent model provided in the embodiment of the present application.

[0046] An embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the training method of the multi-agent model provided by the embodiment of the present application.

[0047] The embodiments of the present application have the following beneficial effects:

[0048] Compared with the technology of using the federated averaging algorithm for model training in the related art, the training method of the multi-intelligent model based on horizontal federated learning in the embodiment of the present application enables multi-participant devices to jointly train the multi-agent model under the coordination of the parameter aggregation device, which can ensure the security of the local data of each participant device. At the same time, based on each parameter value group and the loss value of each participant device, a corresponding number of target parameter value groups are obtained, which can better and faster determine the appropriate target parameter value group, and the entire multi-intelligent model update process is more stable. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is an optional architecture diagram of a training system for a multi-agent model provided in an embodiment of the present application;

[0050] Figure 2 is an optional structural diagram of an electronic device provided in an embodiment of the present application;

[0051] Figure 3 It is an optional flowchart of the training method of the multi-agent model provided in the embodiment of the present application;

[0052] Figure 4 It is a comparison diagram of the training process of the multi-intelligence model and the machine learning model provided in the embodiment of the present application;

[0053] Figure 5 It is a schematic diagram of updating a multi-agent model based on a plane triangle provided in an embodiment of the present application;

[0054] Figure 6 This is an optional flow chart for obtaining a target parameter value group provided in an embodiment of the present application;

[0055] Figure 7 is another optional flow chart for obtaining a target parameter value group provided in an embodiment of the present application;

[0056] Figure 8 is a flow chart of a method for selecting a target parameter value group provided in an embodiment of the present application;

[0057] Fig. 9 is a multi-agent model prediction flow chart provided in an embodiment of the present application;

[0058] Fig.10 is a schematic diagram of multi-agent model optimization provided by an embodiment of the present application;

[0059] Fig.11 It is an optimized schematic diagram of a multi-agent model based on horizontal federated learning provided in an embodiment of the present application;

[0060] Fig.12 is a flow chart of a verification method of a multi-agent model provided in an embodiment of the present application;

[0061] Fig.13 It is a flowchart of the verification method of the multi-agent model based on horizontal federated learning provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0063] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0064] If similar descriptions of "first / second" appear in the application documents, the following instructions are added. In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0066] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0067] 1) Federated Learning is a distributed learning machine learning framework that conducts efficient machine learning between computing devices of multiple participants while ensuring information security during data exchange, protecting terminal data and personal data privacy, and ensuring legality and compliance.

[0068] 2) Horizontal Federated Learning, also known as sample-based federated learning, is a method of performing joint machine learning on the data with the same data features but different users when there is a lot of overlap in the data features of the participants but little overlap in the users.

[0069] For example, there are two banks in different regions. Their user groups come from their respective regions and have little overlap. However, their businesses are very similar, and the characteristics of the recorded user data are largely the same. Horizontal federated learning can be used to help the two banks build a joint model to predict their customer behavior.

[0070] 3) Agent-based simulation or agent-based modeling (ABS or ABM): A computational model used to simulate the actions and interactions of agents (independent individuals or collective groups, such as organizations and teams). A multi-agent model is a microscopic model that reproduces and predicts complex phenomena by simulating the simultaneous actions and interactions of multiple agents. This process is an emergence from a low (micro) level to a high (macro) level.

[0071] According to ABS, urban traffic conditions and disease transmission can be simulated. For example, the spread of the new coronavirus can be simulated through ABS to help predict the development of the new coronavirus epidemic and analyze the inhibitory effects of different intervention measures on the epidemic. In this scenario, three parts are usually involved: 1) a population model close to the real distribution; 2) a social network model between people; 3) a disease transmission model; based on the above three models and corresponding parameters, the development trend of the epidemic can be simulated given the initial number of infections.

[0072] 4) Nelder-Mead (NM) is used to find the maximum and minimum values ​​of the objective function in multidimensional space. It is a direct search method based on comparison and is usually applied to nonlinear optimization problems where the derivative is unknown to solve the objective function minimization problem.

[0073] The n-dimensional Nelder–Mead includes n+1 test points, forming a simplex, and then calculating the objective function value of each point, with the goal of finding a new test point to replace the old test point, and iterating the process. The simplest method is to replace the worst point with the reflection point of the centroid (the mean of the first n points). If the reflection point is better than the current point, you can continue to extend the search in the direction of the reflection point; if it is not as good as the current point, then shrink all the points to a better direction.

[0074] The embodiments of the present application provide a training method, device, electronic device, computer-readable storage medium and computer program product for a multi-agent model, which can find suitable parameter values ​​better and faster, making the model optimization process more stable and controllable.

[0075] Based on the above explanations of the nouns and terms involved in the embodiments of the present application, the training system of the multi-agent model provided in the embodiments of the present application is first described. Figure 1 , Figure 1 It is an optional architectural diagram of a training system for a multi-agent model provided in an embodiment of the present application. In the training system 100 for a multi-agent model, a participant device 400 is included (two participant devices are shown as an example, respectively denoted as 400-1 and 400-2, for distinction. In actual implementation, multiple participant devices may be included). The participant device 400-1 and the participant device 400-2 are interconnected via a network 300 and are connected to a parameter aggregation device 200 via the network 300. The network 300 may be a wide area network or a local area network, or a combination of the two, and data transmission is achieved using a wireless link.

[0076] In some embodiments, the participant device can be a laptop, a tablet computer, a desktop computer, a smart phone, a dedicated messaging device, a portable gaming device, a smart speaker, a smart watch, etc., and can also be a client terminal of a federated learning participant, such as a participant device in each region that stores user feature data, but is not limited to this. The parameter aggregation device 200 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDN, Content Delivery Network), and big data and artificial intelligence platforms, etc., to assist each participant device in performing federated learning to obtain a federated learning model. The network 300 can be a wide area network or a local area network, or a combination of the two. The first participant device and the parameter aggregation device can be directly or indirectly connected by wired or wireless communication, which is not limited in the embodiments of the present application.

[0077] The participant device 400 is used to train the local multi-agent model and send a first number of parameter value groups obtained from the training of the local multi-agent model to a parameter aggregation device; receive a first number of target parameter value groups after reselection sent by the parameter aggregation device, and update the local multi-agent model based on the obtained first number of target parameter value groups.

[0078] The parameter aggregation device 200 is used to receive a first number of parameter value groups sent by each participating device, the parameter value group including a parameter value of at least one unpredictable parameter; summarize the parameter value groups sent by each participating device, and send a second number of parameter value groups obtained by summarizing to each participating device respectively; receive a second number of loss values ​​returned by each participating device, the loss value is determined by the participating device based on the received parameter value group and the loss function of the multi-agent model; based on the second number of loss values, select a first number of parameter value groups from the second number of parameter value groups as target parameter value groups; send the target parameter value groups to each participating device respectively, so that each participating device updates the model parameters of the multi-agent model based on the target parameter value groups.

[0079] See also Figure 2 , Figure 2 is an optional structural diagram of an electronic device 500 provided in an embodiment of the present application. In practical applications, the electronic device 500 can be implemented as Figure 1 The participant device 400 or parameter aggregation device 200 in the embodiment of the present invention is an electronic device. Figure 1 Taking the parameter aggregation device 200 shown as an example, an electronic device for implementing the training method of the multi-agent model of an embodiment of the present application is described. Figure 2 The electronic device 500 shown includes: at least one processor 510, a memory 550, at least one network interface 520 and a user interface 530. The various components in the electronic device 500 are coupled together via a bus system 540. It can be understood that the bus system 540 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 540 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 540 is not described in detail. Figure 2 Various buses are labeled as bus system 540 .

[0080] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0081] The user interface 530 includes one or more output devices 531 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0082] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 550 may optionally include one or more storage devices that are physically remote from the processor 510.

[0083] The memory 550 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 550 described in the embodiments of the present application is intended to include any suitable type of memory.

[0084] In some embodiments, the memory 550 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0085] Operating system 551, including system programs for processing various basic system services and performing hardware-related tasks, such as framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0086] A network communication module 552, for reaching other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB);

[0087] a presentation module 553 for enabling presentation of information via one or more output devices 531 (e.g., display screen, speaker, etc.) associated with the user interface 530 (e.g., a user interface for operating peripherals and displaying content and information);

[0088] The input processing module 554 is used to detect one or more user inputs or interactions from one of the one or more input devices 532 and translate the detected inputs or interactions.

[0089] In some embodiments, the training device of the multi-agent model provided in the embodiments of the present application can be implemented in software. Figure 2A training device 555 of a multi-agent model stored in a memory 550 is shown, which may be software in the form of a program or a plug-in, and includes the following software modules: a first receiving module 5551, a summarizing module 5552, a second receiving module 5553, a selecting module 5554, and a sending module 5555. These modules are logical, and thus may be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0090] In other embodiments, the training device of the multi-agent model provided in the embodiments of the present application can be implemented in hardware. As an example, the training device of the multi-agent model provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the training method of the multi-agent model provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs) or other electronic components.

[0091] The training method of the multi-agent model provided by the embodiment of the present application will be described in combination with the exemplary application and implementation of the parameter aggregation device provided by the embodiment of the present application. Figure 3 , Figure 3 is a flow chart of the training method of the multi-agent model provided in the embodiment of the present application, which will be combined with Figure 3 The steps shown are explained.

[0092] In step 101, a parameter aggregation device receives a first number of parameter value groups sent by each participant device, where the parameter value groups include a parameter value of at least one unpredictable parameter.

[0093] Here, the parameter value group received by the parameter aggregation device may be the first number of parameter value groups after each participant device initializes the parameters of the local multi-agent model, or may be the first number of parameter value groups obtained after each participant device performs a preset round of iterative updates on the local multi-agent model. It should be noted that there is a quantitative relationship between the first number and the number of unpredictable parameters included in the multi-agent model.

[0094] The multi-agent model is explained. The multi-agent model is also called the multi-party agent model. The multi-agent model includes models and parameters. The model refers to each agent. The parameters are used to define specific attributes in the model. The parameters include predictable parameters and unpredictable parameters. Among them, predictable parameters are parameters obtained through historical data and empirical parameters. The values ​​of unpredictable parameters cannot be determined and need to be verified on real data.

[0095] See also Figure 4 , Figure 4 It is a comparison diagram of the training process of the multi-intelligence model and the machine learning model provided in the embodiment of the present application. Obtaining an updated multi-agent model includes constructing an initial multi-agent model (constructing a model), verifying the multi-agent model (verification process) and testing the multi-agent model (testing process). Among them, constructing an initial multi-agent model refers to initializing model parameters, presetting loss functions (used to update the multi-agent model), etc.; the verification process refers to updating unpredictable parameters in the model through preset rounds of iterations; the testing process refers to testing the correctness of the multi-agent model by modifying the output results of the model. Obtaining a converged machine learning model includes constructing an initial machine learning model, training a machine learning model and testing a machine learning model, wherein the training stage of the machine learning model is to iteratively update the machine learning model through training sample data for prediction rounds. It should be noted that the verification process of the multi-agent model on real data is similar to the training process in machine learning, that is, optimizing the value of unpredictable parameters so that the results predicted by the model are as close as possible to the real data.

[0096] In some embodiments, when verifying a multi-agent model, the quantitative relationship between the number of initialized parameter value groups, i.e., the first number, and the number of unpredictable parameters is related to the optimization algorithm set when verifying the multi-agent model.

[0097] In actual implementation, when the NM optimization method is adopted, according to the characteristics of the NM algorithm, assuming that there are q (q≥1 and q is an integer) unpredictable parameters in the multi-intelligent model that need to be fitted and optimized, then each participating device will maintain a combination of n+1 groups of unpredictable parameters during the initialization and model update process, that is, for participating device 1, the dimension of its unpredictable parameter X1,V is q, then when the parameters are initialized, there will be q+1 (that is, the second number is q+1) groups of X1,V with different values.

[0098] Assume that in the multi-agent model f, when the number of unpredictable parameters q = 1, according to the NM optimization algorithm, when the model is verified, the number of parameter value groups initialized is 2. The test points corresponding to the two groups of parameter value groups form a line segment, and the worst point in the line segment (that is, the maximum point of f) is replaced by a new test point. The process of obtaining new test points is iterated to update the length of the line segment and try to make the length of the line segment close to zero. Even if the output result (predicted value) of f is close to the true value, in the process of continuously obtaining new test points, it can be regarded as translating, truncating, and other operations on the line segment with the remaining point other than the worst point as the center. When the number of unpredictable parameters q=2, according to the NM optimization algorithm, during model verification, the number of initialized parameter value groups is 3. The three test points corresponding to the three parameter value groups constitute a plane triangle, and the worst point in the triangle (that is, the maximum point of f) is continuously replaced by a new test point. The process of iteratively obtaining new test points is to continuously transform the triangle (continuously search for the optimal point) so that the area of ​​the triangle continues to decrease until the area of ​​the triangle converges to a point (three vertices overlap into a point), at which time the optimal point is obtained. In the process of continuously obtaining new test points, it can be regarded as replacing the worst point with the reflection point (reflection) of the mean (center point) of the first two points except for the worst point. If the predicted value of f corresponding to the reflection point is better than the predicted value of f corresponding to the current worst point, the search can continue to extend in the direction of the reflection point; if it is worse than the predicted value of f corresponding to the current worst point, all points will be shrunk in a better direction. When the number of unpredictable parameters q=3, according to the NM optimization algorithm, during model verification, the number of parameter value groups initialized is 4. If it is in three-dimensional space, it can be regarded as the smallest volume of the tetrahedron. If it is in plane space, it can be regarded as the smallest area of ​​the quadrilateral. The implementation method is similar to that when q=3. During the entire iteration process, it is sufficient to ensure that the number of parameter value groups is greater than the number of unpredictable parameters.

[0099] For example, see Figure 5 , Figure 5 Schematic diagram of updating a multi-agent model based on a plane triangle provided in an embodiment of the present application, such as Figure 5 As shown, the multi-agent model f contains two unpredictable parameters p1 and p2. Based on the characteristics of the NM algorithm, three groups of parameter values ​​are constructed, corresponding to points a, b, and c, respectively. It can be seen that f(b)<f(a)<f(c), so the value corresponding to point c is the maximum value, that is, point c is the worst point. Based on the center point o of the remaining two points a and b, the reflection point e of point c relative to point o is obtained. At this time, it can be determined that f(e)<f(b)<f(a), indicating that the optimal point g that can make the f value smaller and smaller can be continued in the direction of the reflection point e.

[0100] In step 102, the parameter value groups sent by each participant device are aggregated, and a second number of parameter value groups obtained by aggregation are respectively sent to each participant device.

[0101] In some embodiments, the parameter aggregation device receives the parameter value groups uploaded by each participant device, and sends the first number of parameter groups corresponding to each participant device as a whole to each participant separately, which can increase the data volume.

[0102] In actual implementation, it is assumed that the multi-agent model based on horizontal federated learning includes K participants, K>1 and K is an integer, and the local multi-agent model structure of each participant device is the same, and each participant device uploads a first number of parameter value groups, assuming that the first number is n+1 (n≥1 and n is an integer), n is the number of unpredictable parameters contained in the multi-agent model, then the parameter aggregation device obtains a second number of parameter value groups, where the second number is K*(n+1). The obtained K*(n+1) groups of parameter value groups are sent to each participant respectively, and at this time, each participant device obtains K*(n+1) groups of parameter value groups.

[0103] In step 103, a second number of loss values ​​returned by each participant device is received, where the loss value is determined by the participant device based on the received parameter value group and the loss function of the multi-agent model.

[0104] Here, each participating device obtains a second number of parameter group values ​​from the parameter aggregation device, and determines the predicted value corresponding to each parameter value group in the second number of parameter value groups based on the local multi-agent model, that is, each participating device will obtain the second number of predicted values, and based on the loss function preset by the local multi-agent model, determine the error between the predicted value corresponding to each parameter value group and the true value corresponding to each parameter value group, that is, the loss value, and the number of loss values ​​is equal to the second number. It can be understood that the number of parameter value groups, predicted values ​​and loss values ​​are equal.

[0105] The setting method of the loss function of the multi-agent model is described. In some embodiments, the loss function is determined according to the form of the true value of the fit. When the true value of the fit is in numerical form, the loss function can be the mean square error between the true value and the predicted value. In other embodiments, when the true value of the fit is in the form of a curve, the mean square error of each point can be summed or maximized.

[0106] For example, the mean squared error (MSE) between the predicted value and the true value is used as the loss function L, and the optimization goal is to find a set of parameters that minimize the value of L. The formula for L is as follows:

[0107] (1)

[0108] In formula (1), Represents the error between the true value and the predicted value.

[0109] In step 104, based on the loss values ​​of the second numbers, the first number of parameter value groups are selected from the second number of parameter value groups as target parameter value groups.

[0110] Here, each participating device uploads the second number of loss values ​​determined locally to the parameter aggregation device. The parameter aggregation device uses each parameter value group in the second number of parameter value groups as an index, performs statistical operations on the loss values ​​corresponding to each parameter value group in each participating device, and selects the first number of parameter value groups from the second number of parameter value groups as the target parameter value groups.

[0111] Continuing with the above example, the K participating devices respectively determine the loss values ​​of the K*(n+1) parameter value groups they obtain, and upload them to the parameter aggregation device respectively. At this time, there are a total of K*K*(n+1) loss values ​​in the parameter aggregation device. It can be understood that each parameter value group in the K*(n+1) parameter value groups will correspond to K loss values ​​(each parameter value group will correspond to each participating device and a loss value will be obtained. Since there are K participating devices, each parameter value group in the parameter aggregation device will correspond to K loss values). The K loss values ​​corresponding to each parameter value group are counted according to a preset statistical method to obtain a reference loss value (or statistical value) corresponding to each parameter value group. The reference loss value is used to characterize the influence of the parameter value group on the multi-agent model, and based on the obtained reference loss value, a first number of target parameter value groups are obtained from a second number of parameter value groups.

[0112] In some embodiments, the parameter value group is used as an index to obtain the loss value of each participant device corresponding to the parameter value group, and multiple loss values ​​are summed to obtain a total loss value as a reference loss value for the current parameter value group. In some embodiments, the parameter value group is used as an index to obtain the loss value of each participant device corresponding to the parameter value group, and multiple loss values ​​are averaged to obtain a loss mean as a reference loss value for the current parameter value group. The embodiments of the present application do not limit the method for obtaining statistical values.

[0113] Exemplarily, among the K*(n+1) groups of parameter value groups, each parameter value group corresponds to K loss values, and the total loss value is obtained by summing these K loss values, such as the total loss value corresponding to the j-th group of parameter value groups is recorded as sum(L 1,j +…+LK,j ), and the total loss value is used as the reference loss value corresponding to the parameter value group; the K loss values ​​corresponding to each parameter value group can also be averaged to obtain the mean loss value, such as the j-th group of parameter value groups, the corresponding mean loss value is recorded as avg(L 1,j +…+LK,j ), and the mean loss value is used as the reference loss value corresponding to the parameter value group.

[0114] The specific implementation method of obtaining the target parameter value group is described. In some embodiments, the parameter aggregation device performs statistical operations on the second number of loss values ​​determined locally by each participating device, obtains reference loss values ​​corresponding to the parameter value groups one by one, and sorts the reference loss values ​​corresponding to all parameter value groups in descending order, and obtains the parameter value group corresponding to the first number of reference loss values ​​ranked high as the target parameter value group. Figure 6 , Figure 6 is an optional flow chart for obtaining a target parameter value group provided in an embodiment of the present application, based on Figure 3 , combined with Figure 6 The steps shown illustrate step 104 .

[0115] Step 1041a: The parameter aggregation device determines a third number of loss values ​​corresponding to each parameter value group in the second number of parameter value groups based on the second number of loss values ​​returned by each participant device, where the third number is the number of participant devices.

[0116] In combination with the above step 103, it can be known that the parameter aggregation device receives, and each participating device determines the loss value corresponding to each parameter value group in the second number of parameter value groups according to the local multi-agent model and the corresponding loss function.

[0117] For the second number of parameter value groups, if each participating device uploads the second number of loss values ​​respectively, each parameter value group will have a third number of loss values, and the third number is the number of participating devices.

[0118] Exemplarily, {parameter value group 1, loss value a1, loss value a2, loss value a3, …, loss value aN}, {parameter value group 2, loss value b1, loss value b2, loss value b3, …, loss value bN}, …, {parameter value group M, loss value m1, loss value m2, loss value m3, …, loss value mN}, wherein M is the second number, M≥1, N is the third number, N≥1, loss value a1, loss value b1, …, loss value m1 comes from participant device 1, loss value a2, loss value b2, …, loss value m2 comes from participant device 2, and so on, loss value aN, loss value bN, …, loss value mN comes from participant device N.

[0119] Step 1042a: Determine a reference loss value corresponding to each parameter value group based on the third number of loss values ​​corresponding to each parameter value group.

[0120] In some embodiments, the reference loss value corresponding to each parameter value group can be obtained by summing: the following processing is performed on each parameter value group in the second number of parameter value groups: the loss values ​​of the third number corresponding to the parameter value group are summed to obtain a summed result; and the summed result is used as the reference loss value corresponding to the parameter value group.

[0121] In some embodiments, the reference loss value corresponding to each parameter value group can also be obtained by taking the average value, and the following processing is performed on each parameter value group in the second number of parameter value groups: the third number of loss values ​​corresponding to the parameter value group are averaged to obtain the loss mean; and the loss mean is used as the reference loss value corresponding to the parameter value group.

[0122] Continuing with the above example, taking the reference loss value corresponding to parameter value group 1 as an example, the reference loss value 1 corresponding to parameter value group 1 is obtained by summing {parameter value group 1, loss value a1, loss value a2, loss value a3, ..., loss value aN}, that is, reference loss value 1 = loss value a1 + ... + loss value aN; the reference loss value 1 corresponding to parameter value group 1 is obtained by averaging, that is, reference loss value 1 = (loss value a1 + ... + loss value aN) / N. Finally, the reference loss values ​​corresponding to each parameter value group are {parameter value group 1, reference loss value 1}, {parameter value group 2, reference loss value 2}, ..., {parameter value group M, reference loss value M}.

[0123] Step 1043a, sorting the second number of parameter value groups according to the reference loss values ​​from small to large, and selecting from the first parameter value group according to the sorting result until the first number of parameter value groups are selected as the target parameter value groups.

[0124] Here, the smaller the reference loss value is, the smaller the error between the predicted value of the multi-intelligence model and the actual value (or true value) is, and the more stable the corresponding parameter value is.

[0125] Continuing with the above example, the reference loss values ​​corresponding to the second number of parameter value groups (the second number is M), i.e., {parameter value group 1, reference loss value 1}, {parameter value group 2, reference loss value 2}, ..., {parameter value group M, reference loss value M}, are sorted in descending order according to the size of the reference loss values, and the first parameter value group is selected based on the sorting result until the first number of parameter value groups are selected as the target parameter value groups.

[0126] Continuing to explain the specific implementation method of obtaining the target parameter value group, in some embodiments, the target parameter value group can also be selected based on the sorting of the loss values ​​of each parameter value group corresponding to each participant device. The parameter aggregation device sorts the second number of loss values ​​determined locally for each participant device according to the size of the loss value, and then obtains the first number of target parameter value groups based on the sorted sequence number. Figure 7 , Figure 7 is an optional flow chart for obtaining a target parameter value group provided in an embodiment of the present application, based on Figure 3 , combined with Figure 7 The steps shown illustrate step 104 .

[0127] Step 1041b: The parameter aggregation device sorts the second number of loss values ​​returned by each participant device respectively to obtain a loss value sequence corresponding to each participant device.

[0128] In some embodiments, the target parameter value group can be determined based on the average ranking of the loss values. The parameter aggregation device sorts the loss values ​​of each participant device corresponding to each parameter value group in ascending order to obtain a corresponding sequence of numbers (i.e., the elements in the sequence represent the ranking corresponding to the loss values). Figure 8 , Figure 8 is a flow chart of a method for selecting a target parameter value group provided in an embodiment of the present application. Step 1041b can be combined with Figure 8 The steps shown are explained.

[0129] Step 201: The parameter aggregation device sorts the second number of loss values ​​returned by each participant device respectively to obtain a loss value sequence corresponding to each participant device.

[0130] Exemplarily, among K*(n+1) groups of parameter value groups, each parameter value group corresponds to K loss values, and these K loss values ​​are sorted in ascending order. For example, the j-th group of parameter value groups, j=1 and j≤K*(n+1), corresponds to K participating devices, and K ranking ranks (serial numbers) will be obtained, recorded as N1,j,...,NK,j.

[0131] In step 202, the parameter aggregation device obtains the order of the loss values ​​corresponding to each parameter value group in each loss value sequence.

[0132] Step 203: The parameter aggregation device selects a first number of parameter value groups from a second number of parameter value groups as target parameter value groups based on the order of the parameter value groups in the loss value sequences.

[0133] In some embodiments, obtaining a target parameter value group based on the ranking of each parameter value group in each loss value sequence can be achieved in the following manner: the parameter aggregation device averages the ranking of the parameter value group in each loss value sequence for each parameter value group to obtain a ranking average value; based on the ranking average value of each parameter value group in each loss value sequence, a first number of parameter value groups are selected from a second number of parameter value groups as the target parameter value groups.

[0134] In actual implementation, the sorting numbers (sorting indexes) corresponding to the current parameter value group in each loss value sequence are averaged to obtain the sorting average value (sequence number mean) corresponding to the current parameter value group. In ascending order, the obtained sequence number means are sorted to obtain the first number of sequence number means with the highest ranking, and the corresponding parameter value group is determined as the target parameter value group.

[0135] Exemplarily, among the K*(n+1) groups of parameter value groups, each parameter value group corresponds to K ranking ranks (serial numbers), N 1,j , ..., NK,j , and the obtained K rankings are averaged to obtain the serial number mean, which is recorded as avg(N 1,j+...+NK,j ).

[0136] In some embodiments, obtaining a target parameter value group based on the ranking of each parameter value group in each loss value sequence can also be achieved in the following way: the parameter aggregation device sums the ranking of each parameter value group in each loss value sequence to obtain the sum of the rankings; and selecting a first number of parameter value groups from a second number of parameter value groups as the target parameter value group based on the sum of the rankings of each parameter value group in each loss value sequence.

[0137] Exemplarily, among the K*(n+1) groups of parameter value groups, each parameter value group corresponds to K ranking ranks (serial numbers), N 1,j , ..., NK,j , and the obtained K rankings are summed to obtain the total serial number value, which is recorded as sum(N 1,j + ... + NK,j ).

[0138] Step 1042b: The parameter aggregation device obtains the order of the loss values ​​corresponding to each parameter value group in each loss value sequence.

[0139] Continuing with the above example, the K*(n+1) sorting means (sequence number means) or sorting total values ​​(sequence number total values) corresponding to the K*(n+1) groups of parameter value groups are sorted in ascending order.

[0140] Step 1043b: The parameter aggregation device selects a first number of parameter value groups from a second number of parameter value groups as target parameter value groups based on the order of each parameter value group in each loss value sequence.

[0141] Continuing with the above example, the parameter aggregation device obtains the first (n+1) statistical values ​​as candidate statistical values, and obtains the parameter value groups corresponding to the candidate statistical values ​​as the target parameter value groups, that is, the first number (n+1) groups of parameter value groups with small errors are finally selected as the target parameter value groups.

[0142] In step 105, the target parameter value group is sent to each of the participant devices respectively, so that each of the participant devices updates the model parameters of the multi-agent model based on the target parameter value group.

[0143] Here, the parameter aggregation device sends the first number of parameter value groups selected in step 104 to each participating device. Each participating device updates (optimizes) the model parameters of the local multi-agent model based on the target parameter value group, which can accelerate the convergence speed of the multi-intelligence model.

[0144] Repeat steps 101 to 105 to obtain an updated multi-agent model. In some embodiments, the parameter aggregation device or the participant device can use the updated multi-agent model to perform information prediction. It should be noted that the parameter aggregation device can be an independent server device or one of the participant devices. Fig. 9 , Fig. 9 This is a multi-agent model prediction flow chart provided in the embodiment of the present application, combined with Fig. 9 The steps shown are explained.

[0145] Step 301: The participant device obtains the predictable parameters and unpredictable parameters corresponding to the updated multi-agent model.

[0146] In some embodiments, it can be used to predict the spread of a disease, that is, to predict the number of confirmed cases and deaths of a target disease and the corresponding spread coefficient in the future. Based on this, the predictable parameters include the population distribution corresponding to the target area, and the unpredictable parameters include the spread coefficient corresponding to the target disease.

[0147] In some embodiments, the updated multi-agent model can be used to predict urban traffic conditions, that is, to predict the number of congested vehicles and the corresponding congestion coefficient for the target road section in the target area within a target time period in the future. Based on this, predictable parameters may include the travel trajectory of the population in the target area, the distribution of office areas, holiday times and weather conditions in the target area, etc., and unpredictable parameters include the congestion coefficient of the target time period.

[0148] Step 302: Initialize the predictable parameters and the unpredictable parameters to obtain a fourth number of initial parameter value groups.

[0149] Continuing with the above disease prediction example, the values ​​of the fourth quantity corresponding to the population distribution and the values ​​of the fourth quantity corresponding to the propagation coefficient are initialized respectively to obtain an initial parameter value group of the fourth quantity.

[0150] Continuing with the above-mentioned example of urban traffic condition prediction, the values ​​of the fourth quantity corresponding to the population travel trajectory, office area distribution, holiday time and weather conditions in the target area, as well as the value of the fourth quantity corresponding to the congestion coefficient are initialized respectively to obtain an initial parameter value group of the fourth quantity.

[0151] Step 303: input a fourth number of initial parameter value groups into the multi-agent model to obtain a prediction result.

[0152] Continuing with the above disease prediction example, the fourth number of initial parameter value groups is input into the updated multi-agent model to obtain the number of confirmed cases and deaths corresponding to the target disease.

[0153] Continuing with the above-mentioned urban traffic situation prediction example, the fourth number of initial parameter value groups are input into the updated multi-agent model to obtain the number of congested vehicles on the target road section in the target area within the target time period.

[0154] Step 304: based on the prediction result, update the unpredictable parameter to obtain the target value corresponding to the unpredictable parameter.

[0155] Continuing with the above example, based on the number of confirmed cases and the number of deaths, as well as the optimization algorithm, the transmission coefficient corresponding to the target disease is determined.

[0156] Continuing with the above-mentioned urban traffic situation prediction example, the congestion coefficient corresponding to the target road section is determined based on the number of congested vehicles on the target road section during the target time period and the optimization algorithm.

[0157] The training method of the multi-agent model based on horizontal federated learning provided in the embodiment of the present application is as follows: the parameter aggregation device receives the first number of parameter value groups sent by each participating device, and the parameter value group includes the parameter value of at least one unpredictable parameter; the parameter value groups sent by each participating device are summarized, and the second number of parameter value groups obtained by the summary are sent to each participating device respectively, which can increase the amount of verification data, improve the verification efficiency, and solve the problem of data islands of each participating party in the multi-agent model; the second number of loss values ​​returned by each participating device are received; based on the second number of loss values, the first number of parameter value groups are selected from the second number of parameter value groups as the target parameter value group, which can effectively eliminate the impact of the difference in loss values ​​of different participants; the target parameter value group is sent to each participating device respectively, so that each participating device updates the model parameters of the multi-agent model based on the target parameter value group. Based on this, the local data security of each participating device can be guaranteed, the problem of data islands in the field of multi-agent models can be solved, and the convergence speed of multi-intelligence models can be accelerated.

[0158] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.

[0159] As the COVID-19 outbreak spreads in countries and regions around the world, one of the main issues facing governments is how to formulate effective strategies to deal with it. If we can understand the future spread of the epidemic and the effectiveness of the implementation of various measures, we can formulate more reasonable policies in a targeted manner. One way to understand this kind of information is through multi-agent model simulation, which predicts the spread of the epidemic in the region by modeling the population and population activities in the corresponding region.

[0160] A constructed ABS model can be applied to different regions. It only needs to adjust its predictable parameters (such as the age and gender ratio of the population) according to the corresponding situation in the target area, and then verify the value of the unpredictable parameters. The model can be used to predict and analyze the subsequent development of the epidemic in the target area. Generally, the larger the area involved in the simulation and the more agents used, the better the model effect and the more accurately it can reflect the real situation of the system. However, due to the population distribution and population activity in various regions, the epidemic situation data may involve privacy or security issues and are relatively sensitive. These data are usually only accessible to local credit institutions and cannot be aggregated in one place for training / verification. Therefore, each institution can only perform verification simulations based on its own limited data. The values ​​of unpredictable parameters obtained are often not the optimal results, the model effect will be affected, and ultimately lead to prediction deviations.

[0161] The goal of horizontal federated learning is to use the limited data of all parties to jointly train a global model while ensuring data security. Because the global model uses the data of each participant for training, its model effect can be close to the situation where the data of each participant is trained together, and is significantly better than the effect of the model obtained by each participant based only on its own data. However, the use of multi-agent models is very different from traditional machine learning. How to apply federated learning to solve the verification (training) of multi-agent models involving multiple parties is an unresolved problem.

[0162] The simulation of multi-agent models usually includes two parts: model and parameters. The model defines the operating rules of each agent in the model (such as pre-defined rules, such as how the population is distributed, how to form a family, and the usual travel rules, such as going to work, school, shopping, etc., in the simulation of the COVID-19 epidemic in a region, and under what circumstances the disease will spread). The parameters define the specific attributes in the model, which are divided into predictable parameters and unpredictable parameters. Predictable parameters can be obtained from sources such as public data, literature or experience (for example, when constructing a population, how the gender and age ratio of the population is composed, the distribution of the number of people in the family, how many days a week people of different ages and working status go out, and what means of transportation they use when going out). Through these parameters, the model population and corresponding attributes and preferences (such as travel preferences) can be constructed based on a specific probability model. In contrast, unpredictable parameters cannot be obtained from the above channels (such as the transmission coefficient in the formula of the disease transmission law), and need to be optimized based on the model prediction output value and the true value. The goal is to obtain the optimal unpredictable parameter value so that the model prediction is as close to the true value as possible. This process is similar to the model training process of conventional machine learning.

[0163] Taking the above-mentioned propagation coefficient as an example, the model builds a simulation model adapted to the region based on the population characteristics of a certain region. Based on the ABS model, the daily number of confirmed cases and deaths in the region over a period of time can be output; these data can also be obtained from public channels as real data (equivalent to Y_groundtruth in machine learning). The input of the ABS model is the already obtained predictable parameters X_fixed (including population distribution, etc.), unpredictable parameter values ​​X_variable (here, because only one parameter - the propagation coefficient is optimized, so n = 1, NM optimization uses n+1 sets of values), and the output of the model is the number of confirmed cases and deaths in a certain region over a period of time. Through the Nelder-Mead optimization algorithm, an optimal value of the propagation coefficient (X_variable) can be optimized based on Y_groundtruth and model output, which can make the output of the model closest to the true value.

[0164] When the update (optimization, verification) of the multi-agent model is completed, all the parameters contained in the multi-agent have been determined. In practical applications, the correctness of the multi-agent model can be tested on other output indicators (for example, the number of confirmed cases and deaths in another period of time). After passing the test, other situations of interest can be simulated by changing the model conditions, such as simulating urban traffic conditions.

[0165] In some embodiments, the multi-agent model can be executed separately on the relevant electronic device, see Fig.10 , Fig.10 is a schematic diagram of multi-agent model optimization provided by an embodiment of the present application,

[0166] In some embodiments, the multi-agent model can be applied to a horizontal federated learning system, where multiple participating devices jointly participate in the optimization operation of the multi-agent model, see Fig.11 , Fig.11 is an optimized schematic diagram of a multi-agent model based on horizontal federated learning provided in an embodiment of the present application, Fig.11 The optimization method of the multi-agent model applied to the horizontal federated learning system is demonstrated by taking Nelder-Mead Optimization as an example. It should be noted that for the convenience of description, Fig.11 Only the server and one participant are shown in the figure. In actual application scenarios, there may be multiple participants, denoted as Client 1, ..., Client K, where K is a positive integer and K≥1. The structure and working mode of the local multi-agent model contained in each participant are the same as those of participant 1 (Client 1). That is, each participant contains the following same information: 1) There is an identical ABS model f; 2) There are their own private predictable parameters, X 1,E, ..., XN,E, whose values ​​are determined according to the local conditions of each party (corresponding to the private data in the machine learning model); 3) Their own unpredictable parameters, X 1,V, ..., XN,V, whose values ​​cannot be deduced from existing data or experience, and need to be jointly calculated and determined through the values ​​to be simulated (corresponding to the model parameters of each party in machine learning); 4) The target variables Y 1,gt, ..., YN,gt simulated by the local multi-agent model of each participant (corresponding to the labels in machine learning training). There are two other parts, namely the loss function module for calculating the simulation results Y pred and Y gt , a typical function is the mean square error (Y pred -Y gt ) 2 , and the optimization module for optimizing the unpredictable parameters, which is determined by the optimization method used. The server (also called the parameter aggregation device) has an aggregation module for aggregating the values ​​of the unpredictable parameters uploaded by each party after local update.

[0167] It should be noted that the prerequisite for executing multi-agent model verification (optimization) is: according to the characteristics of the NM optimization method, assuming that there are n unpredictable parameters that need to be fitted and optimized, then each participant maintains n+1 sets of unpredictable parameter combinations during the optimization initialization and process, that is, for participant 1, the dimension of its unpredictable parameter X1,V is n, then during initialization, it has n+1 sets of X1,V with different values.

[0168] Next, the process of local iterative verification of the multi-agent model by each participant is described, see Fig.12 , Fig.12 is a flow chart of the verification method of the multi-agent model provided in the embodiment of the present application, combined with Fig.10 as well as Fig.12 The steps shown are explained.

[0169] In step 401, the participants input each set of parameter values ​​into the multi-agent model for simulation to obtain simulation results.

[0170] Here, each participant initializes the unpredictable parameters related to the local multi-agent model (the number of unpredictable parameters is set to n), and each participant obtains n+1 sets of unpredictable parameter value groups, recorded as (Xk, V1, ..., Xk, Vn+1), where k = 1, ..., K, K is a positive integer greater than 1, and K is used to represent the number of participating devices.

[0171] For example, participant k uses each set of parameter values ​​through the multi-agent model f to calculate the simulation result Y, which is recorded as Y k,pred1 ,…,Y k,predn+1 .

[0172] In step 402, the participants determine the loss value corresponding to each parameter value group based on the simulation results corresponding to each parameter value group and the actual value corresponding to each parameter value group through a preset loss function.

[0173] For example, the simulation results Y k,pred1 , ..., Y k,predn+1 obtained in step 401 are respectively compared with the true values ​​Y k,gt corresponding to each parameter value group through the loss function to obtain the losses L k1 , ..., L kn+1 . The loss function is determined by the form of the true value of the fit. When the true value of the fit is in a numerical form, the loss function can be the mean square error between the predicted value and the true value; when the true value of the fit is in a curve form, the mean square error of each point can be summed or the maximum value can be calculated.

[0174] Step 403 verifies the multi-intelligence model based on the loss values ​​corresponding to each parameter value group and the preset optimization algorithm.

[0175] For example, based on the loss values ​​L k1 , . . . , L kn+1 corresponding to each parameter value group obtained in step 402 , the NM optimization algorithm is used to update the values ​​of the unpredictable parameters in the multi-agent model.

[0176] The above steps 401 to 403 are repeatedly performed until the number of iterations reaches the preset round QL.

[0177] In the embodiment of the present application, each participant iterates the prediction rounds of the local multi-agent model, which can effectively improve the optimization speed of the multi-agent model.

[0178] Next, the basic verification process of the multi-agent model based on horizontal federated learning is described. Fig.13 , Fig.13 is a flow chart of a verification method for a multi-agent model based on horizontal federated learning provided in an embodiment of the present application, combined with Fig.11 as well as Fig.13 The steps shown are explained.

[0179] Step 501: After each participant completes a preset number of iterations of the multi-agent model, the updated local first number of unpredictable parameter value groups are uploaded to the server.

[0180] For example, each participant performs QL iterations on a local multi-agent model containing n unpredictable parameters, and each participant obtains n+1 sets of unpredictable parameter value groups.

[0181] Step 502: The server performs sorting and aggregation based on the unpredictable parameter groups uploaded by the parties to obtain a first number of target unpredictable parameter value groups.

[0182] The server performs sorting and aggregation based on K*(n+1) groups of unpredictable parameters uploaded by each participant (assuming there are K participants in total) to obtain n+1 groups of values ​​of unpredictable parameters.

[0183] For example, a typical implementation is as follows: the server aggregates the unpredictable parameters uploaded by K participants (a total of K*(n+1) groups of parameters) to obtain n+1 groups of parameter values. In order to implement the aggregation process, the K*(n+1) groups of parameters are sent to all participants, each participant calculates the loss function value, and then aggregates to obtain K*K*(n+1) loss values. For the jth group of parameters, the calculation results of K loss functions can be obtained, L 1,j ,…,LK,j , where j=1,…,K*(n+1).

[0184] In practical applications, global optimization can be achieved through a variety of sorting methods to select the optimal value. One method is to calculate the total loss of each group of parameters in each participant, such as the jth group of parameters: L j = sum(L 1,j +…+LK,j ), and then select the n+1 groups of parameters with the lowest total loss as the unpredictable parameters obtained by server aggregation.

[0185] Another sorting method is based on average ranking. For example, considering the differences in loss values ​​of different participants, each group of parameters can be sorted according to the participants (such as k parties). Then for the jth group of parameters, there are K rankings, (ri,j, ..., rk,j), and the average can be obtained by r avg,j = average(ri,j, ..., rk,j); then the highest ranking n+1 group of parameters is selected as the unpredictable parameter after aggregation. This sorting selection method can effectively eliminate the impact of the differences in loss values ​​of different participants.

[0186] Step 503: The server sends a first number of target unpredictable parameter value groups to each participant, so that each participant performs local iteration on the multi-agent model based on the target unpredictable parameter value groups.

[0187] For example, the server sends the new n+1 groups of unpredictable parameter values ​​obtained in step 502 to each participant, so that each participant can perform local iteration based on the obtained n+1 groups of unpredictable parameter values.

[0188] Repeat the above steps 501 to 503 until a preset number of rounds is reached, or the multi-agent model converges, thus completing the verification / optimization process of the unpredictable parameters.

[0189] In actual application scenarios, such as simulating urban traffic conditions and disease transmission simulation prediction, the multi-agent model verified by the embodiments of the present application can be used for prediction research.

[0190] In the embodiment of the present application, the local multi-agent model f of each participant can use the Nelder-Mead optimization algorithm to optimize the parameters of the multi-agent model. Combined with the characteristics of the optimization algorithm, when the multi-agent model f contains n unpredictable parameters, n+1 sets of parameter values ​​(initialization) are constructed for the n unpredictable parameters, thus constituting n+1 sets of parameter values. After each participant performs a preset round of iterative updates on the local multi-agent model (this can speed up the calculation efficiency), in view of the limitation of the local data volume and in order to avoid the overfitting phenomenon of the local model, it is also necessary to combine with other participants to conduct horizontal federated learning (increase the data volume). Based on this, each participant uploads its local n+1 groups of unpredictable parameter values ​​to the server. After the server obtains k*(n+1) groups of parameter values ​​(k represents the number of participants), it selects new n+1 groups of parameter values ​​from these k*(n+1) groups of parameter values ​​through different sorting and aggregation operations. Then, the new n+1 groups of parameter values ​​are sent to each participant as the target parameter value group. Each participant continues to optimize the local model parameters based on the received new n+1 groups of parameter values.

[0191] The horizontal federated learning method based on global optimization provided in the embodiment of the present application enables multiple participants to jointly train a multi-intelligence model under the coordination of the server or the participants, and ensures the security of the local data of each participant, thereby solving the data island problem in the field of multi-intelligence models; by globally sorting the parameter points (such as a method based on average ranking), it is possible to determine the appropriate parameter values ​​better and faster, making the optimization process of the multi-agent model more stable and controllable, and more stable than the method of directly using weighted average of parameter values.

[0192] The following is a description of an exemplary structure of a multi-agent model training device 555 provided in an embodiment of the present application implemented as a software module. In some embodiments, Figure 2 As shown, the software modules stored in the training device 555 of the multi-agent model in the memory 550 may include:

[0193] A first receiving module 5551 is configured to receive a first number of parameter value groups sent by each of the participant devices, wherein the parameter value groups include parameter values ​​of at least two unpredictable parameters;

[0194] A summarizing module 5552 is used to summarize the parameter value groups sent by each of the participant devices, and send the second number of parameter value groups obtained by summarizing to each of the participant devices respectively;

[0195] A second receiving module 5553 is used to receive a second number of loss values ​​returned by each of the participant devices, where the loss value is determined by the participant device based on the received parameter value group and the loss function of the multi-agent model;

[0196] A selection module 5554 is used to select a first number of parameter value groups from the second number of parameter value groups as target parameter value groups based on the second number of loss values;

[0197] The sending module 5555 is used to send the target parameter value group to each of the participating devices respectively, so that each of the participating devices updates the model parameters of the multi-agent model based on the target parameter value group.

[0198] In some embodiments, the selection module is also used to determine a third number of loss values ​​corresponding to each parameter value group in the second number of parameter value groups based on the second number of loss values ​​returned by each of the participating devices, the third number being the number of the participating devices; determine a reference loss value corresponding to each of the parameter value groups based on the third number of loss values ​​corresponding to each of the parameter value groups; sort the second number of parameter value groups in ascending order according to the reference loss values, and start selecting from the first parameter value group according to the sorting result until the first number of parameter value groups are selected as the target parameter value groups.

[0199] In some embodiments, the selection module is also used to perform the following processing for each parameter value group in the second number of parameter value groups: summing the loss values ​​of the third number corresponding to the parameter value group to obtain a summation result; and using the summation result as a reference loss value corresponding to the parameter value group.

[0200] In some embodiments, the selection module is also used to perform the following processing on each parameter value group in the second number of parameter value groups: averaging the third number of loss values ​​corresponding to the parameter value group to obtain a loss mean; and using the loss mean as a reference loss value corresponding to the parameter value group.

[0201] In some embodiments, the selection module is also used to sort the second number of loss values ​​returned by each of the participating devices, respectively, to obtain a loss value sequence corresponding to each participating device; respectively obtain the sorting of the loss values ​​corresponding to each parameter value group in each of the loss value sequences; based on the sorting of each of the parameter value groups in each of the loss value sequences, select a first number of parameter value groups from the second number of parameter value groups as the target parameter value groups.

[0202] In some embodiments, the selection module is also used to, for each of the parameter value groups, respectively average the ranking of the parameter value groups in each of the loss value sequences to obtain a ranking average value; and select a first number of parameter value groups from the second number of parameter value groups as target parameter value groups based on the ranking average values ​​of each of the parameter value groups in each of the loss value sequences.

[0203] In some embodiments, the selection module is further used to, for each of the parameter value groups, sum the rankings of the parameter value groups in each of the loss value sequences to obtain the sum of the rankings; and select a first number of parameter value groups from the second number of parameter value groups as target parameter value groups based on the sum of the rankings of the parameter value groups in each of the loss value sequences.

[0204] In some embodiments, the device also includes an application module, which is used to obtain predictable parameters and unpredictable parameters corresponding to the updated multi-agent model; initialize the predictable parameters and the unpredictable parameters to obtain a fourth number of initial parameter value groups; input the fourth number of initial parameter value groups into the multi-agent model to obtain a prediction result; based on the prediction result, update the unpredictable parameters to obtain a target value corresponding to the unpredictable parameter.

[0205] In some embodiments, the application module is also used to initialize the value of a fourth quantity corresponding to the population distribution and the value of a fourth quantity corresponding to the propagation coefficient to obtain a fourth quantity initial parameter value group; input the fourth quantity initial parameter value group into the updated multi-agent model to obtain the number of confirmed cases and the number of deaths corresponding to the target disease; based on the number of confirmed cases and the number of deaths, update the propagation coefficient to obtain the propagation coefficient corresponding to the target disease.

[0206] It should be noted that the description of the device of the embodiment of the present application is similar to the description of the above-mentioned method embodiment, and has similar beneficial effects as the method embodiment, so it will not be repeated.

[0207] An embodiment of the present application provides a computer program product, including a computer program, characterized in that when the computer program is executed by a processor, the training method of the multi-agent model provided by the embodiment of the present application is implemented.

[0208] The present application embodiment provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will be caused to execute the method provided by the present application embodiment, for example, Figure 3 The training method of the multi-agent model is shown.

[0209] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0210] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0211] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0212] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0213] In summary, the embodiments of the present application can ensure the local data security of each participant's device, solve the data island problem in the field of multi-agent models, and accelerate the convergence speed of multi-intelligent models.

[0214] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A training method for a multi-agent model, characterized in that: Based on a federated learning system, the system includes a parameter aggregation device and at least two participant devices, and the method includes: The parameter aggregation device receives a first number of parameter value groups sent by each of the participant devices, wherein the parameter value groups include a parameter value of at least one unpredictable parameter; Summarizing the parameter value groups sent by each of the participant devices, and sending a second number of parameter value groups obtained by summarizing to each of the participant devices respectively; Receiving a second number of loss values ​​returned by each of the participant devices, wherein the loss values ​​are determined by the participant devices based on the received parameter value group and the loss function of the multi-agent model; Based on a second number of loss values ​​returned by each of the participant devices, selecting a first number of parameter value groups from the second number of parameter value groups as target parameter value groups; The step of selecting the first number of parameter value groups as the target parameter value group from the second number of parameter value groups based on the second number of loss values ​​returned by each of the participant devices specifically includes: the parameter aggregation device uses each parameter value group in the second number of parameter value groups as an index, performs a statistical operation on the loss values ​​corresponding to each of the parameter value groups in each of the participant devices, and selects the first number of parameter value groups as the target parameter value group from the second number of parameter value groups; Sending the target parameter value group to each of the participant devices respectively, so that each of the participant devices updates the model parameters of the multi-agent model based on the target parameter value group; Wherein, the method further comprises: Obtain the predictable and unpredictable parameters corresponding to the updated multi-agent model; Wherein, the predictable parameter includes the population distribution corresponding to the target area, the unpredictable parameter includes the transmission coefficient corresponding to the target disease, and the method further includes: Initializing a fourth number of values ​​corresponding to the population distribution and a fourth number of values ​​corresponding to the propagation coefficient to obtain a fourth number of initial parameter value groups; Inputting the fourth number of initial parameter value groups into the updated multi-agent model to obtain the number of confirmed cases and the number of deaths corresponding to the target disease; Based on the number of confirmed cases and the number of deaths, the transmission coefficient is updated to obtain the transmission coefficient corresponding to the target disease.

2. The method according to claim 1, characterized in that The selecting, based on the second number of loss values ​​returned by each of the participant devices, the first number of parameter value groups from the second number of parameter value groups as the target parameter value groups comprises: Based on the second number of loss values ​​returned by each of the participant devices, determine a third number of loss values ​​corresponding to each parameter value group in the second number of parameter value groups, where the third number is the number of the participant devices; Determine a reference loss value corresponding to each of the parameter value groups based on a third number of loss values ​​corresponding to each of the parameter value groups; The second number of parameter value groups are sorted in ascending order according to the reference loss values, and selection is started from the first parameter value group according to the sorting result until the first number of parameter value groups are selected as the target parameter value groups.

3. The method according to claim 2, characterized in that Determining a reference loss value corresponding to each of the parameter value groups based on a third number of loss values ​​corresponding to each of the parameter value groups includes: The following processing is performed for each parameter value group in the second number of parameter value groups: Summing a third number of loss values ​​corresponding to the parameter value group to obtain a summation result; The summation result is used as the reference loss value corresponding to the parameter value group.

4. The method according to claim 2, characterized in that: Determining a reference loss value corresponding to each parameter value group based on a third number of loss values ​​corresponding to each parameter value group includes: Averaging a third number of loss values ​​corresponding to the parameter value group to obtain a loss mean; The loss mean is used as the reference loss value corresponding to the parameter value group.

5. The method according to claim 1, characterized in that The selecting, based on the second number of loss values ​​returned by each of the participant devices, the first number of parameter value groups from the second number of parameter value groups as the target parameter value groups comprises: respectively sorting the second number of loss values ​​returned by each of the participant devices to obtain a loss value sequence corresponding to each of the participant devices; Respectively obtain the order of the loss values ​​corresponding to each parameter value group in each of the loss value sequences; Based on the order of each of the parameter value groups in each of the loss value sequences, a first number of parameter value groups are selected from the second number of parameter value groups as target parameter value groups.

6. The method according to claim 5, characterized in that The selecting a first number of parameter value groups as target parameter value groups from the second number of parameter value groups based on the order of the parameter value groups in the loss value sequences comprises: For each of the parameter value groups, averaging the ranking of the parameter value group in each of the loss value sequences is performed to obtain a ranking average value; A first number of parameter value groups are selected from the second number of parameter value groups as target parameter value groups according to a ranked average value of each of the parameter value groups in each of the loss value sequences.

7. The method according to claim 5, characterized in that The selecting a first number of parameter value groups as target parameter value groups from the second number of parameter value groups based on the order of the parameter value groups in the loss value sequences comprises: For each of the parameter value groups, summing up the ranking of the parameter value group in each of the loss value sequences to obtain a sum of the rankings; A first number of parameter value groups are selected from the second number of parameter value groups as target parameter value groups according to the sum of the order of the parameter value groups in the loss value sequences.

8. The method according to claim 1, characterized in that After the step of obtaining the predictable parameters and unpredictable parameters corresponding to the updated multi-agent model, the method further includes: Initializing the predictable parameters and the unpredictable parameters to obtain a fourth number of initial parameter value groups; Inputting the fourth number of initial parameter value groups into the multi-agent model to obtain a prediction result; Based on the prediction result, the unpredictable parameter is updated to obtain a target value corresponding to the unpredictable parameter.

9. A training device for a multi-agent model, characterized in that: Based on the federated learning system, the system includes a parameter aggregation device and at least two participant devices, including: A first receiving module, configured to receive a first number of parameter value groups sent by each of the participant devices, wherein the parameter value groups include a parameter value of at least one unpredictable parameter; A summarizing module, used for summarizing the parameter value groups sent by each of the participant devices, and sending the second number of parameter value groups obtained by summarizing to each of the participant devices respectively; A second receiving module, configured to receive a second number of loss values ​​returned by each of the participant devices, wherein the loss values ​​are determined by the participant device based on the received parameter value group and the loss function of the multi-agent model; a selection module, configured to select a first number of parameter value groups from the second number of parameter value groups as target parameter value groups based on the second number of loss values, wherein the selection module is specifically configured for the parameter aggregation device to use each parameter value group in the second number of parameter value groups as an index, perform statistical operations on the loss values ​​corresponding to each parameter value group in each participant device, and select the first number of parameter value groups from the second number of parameter value groups as the target parameter value groups; A sending module, used for sending the target parameter value group to each of the participant devices respectively, so that each of the participant devices updates the model parameters of the multi-agent model based on the target parameter value group; Wherein, the training device of the multi-agent model further comprises: Obtain the predictable and unpredictable parameters corresponding to the updated multi-agent model; The predictable parameters include the population distribution corresponding to the target area, the unpredictable parameters include the propagation coefficient corresponding to the target disease, and the training device of the multi-agent model further includes: Initializing a fourth number of values ​​corresponding to the population distribution and a fourth number of values ​​corresponding to the propagation coefficient to obtain a fourth number of initial parameter value groups; Inputting the fourth number of initial parameter value groups into the updated multi-agent model to obtain the number of confirmed cases and the number of deaths corresponding to the target disease; Based on the number of confirmed cases and the number of deaths, the transmission coefficient is updated to obtain the transmission coefficient corresponding to the target disease.

10. An electronic device, characterized in that: include: A memory for storing executable instructions; A processor, used to implement the multi-agent model training method described in any one of claims 1 to 8 when executing the executable instructions stored in the memory.

11. A computer-readable storage medium, characterized in that: Executable instructions are stored, which are used to implement the training method of the multi-agent model described in any one of claims 1 to 8 when the executable instructions are executed by a processor.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the training method of the multi-agent model described in any one of claims 1 to 8.

Images