Intelligent decision-making method and system based on distributed training framework

Abstract

The present disclosure provides an intelligent decision-making method and system based on a distributed training framework, belonging to the field of artificial intelligence and distributed computing. The distributed training framework includes a sampler, a learner and a coordination server. The intelligent decision-making method comprises: selecting k strategies from a first strategy set as training strategies, and determining the corresponding sampler and learner of each training strategy. Each training strategy is respectively paired with a target strategy as a group for adversarial sampling to obtain first sampling data. The learner is trained based on the first sampling data to obtain network model parameters of the learner, and the network model parameters are aggregated to obtain global model parameters. The global model parameters are sent to each learner and each sampler for iterative training to obtain a target strategy set that meets a preset convergence condition. The present disclosure combines synchronous sampling with distributed parallel computing, making intelligent decision-making applicable to complex military confrontation game scenarios and significantly improving the optimization effect of strategy diversity.

Description

TECHNICAL FIELD

[0001] The present disclosure belongs to the field of artificial intelligence and distributed computing technology, more specifically, it relates to an intelligent decision-making method and system based on a distributed training framework. BACKGROUND

[0002] Military confrontation games can usually be modeled as two-person zero-sum games. In related research, diversity optimization is a key means to improve the robustness and adaptability of agent strategies. In complex military confrontation game scenarios, existing diversity optimization methods usually adjust the reward function, encourage exploration behavior, and other means to achieve optimization, mainly based on state features or state trajectories to optimize at the event level. However, these methods are limited to state and event level adjustments, and have limited effect on overall strategy diversity optimization. Therefore, we need to consider diversity optimization from a more comprehensive strategy level.

[0003] In simple matrix games, some theoretical research has attempted to optimize diversity from the strategy level, known as response diversity. Response diversity helps agents avoid being predicted and exploited by opponents by adjusting the interaction between strategies. However, this method is mainly applied to simple matrix games, and in complex game scenarios, agents cannot make effective decisions. SUMMARY

[0004] The purpose of the present disclosure is to provide an intelligent decision-making method and system based on a distributed training framework to make intelligent decision-making applicable to complex military confrontation game scenarios and significantly improve the optimization effect of strategy diversity.

[0005] In a first aspect of the embodiments of the present disclosure, an intelligent decision-making method based on a distributed training framework is provided, the distributed training framework comprising a sampler, a learner, and a coordination server, and the sampler and the learner are multiple;

[0006] The intelligent decision-making method comprises:

[0007] selecting k strategies from a first strategy set as training strategies, determining a sampler and a learner corresponding to each training strategy; the first strategy set is a strategy set initialized in a game environment;

[0008] conducting adversarial sampling on each training strategy and a target strategy as a group to obtain first sampling data, the target strategy being a strategy in the first strategy set, and the target strategy being different from the k training strategies;

[0009] training the learner based on the first sampling data to obtain network model parameters of the learner, and aggregating the network model parameters to obtain global model parameters;

[0010] The global model parameters are sent to each learner and each sampler for iterative training to obtain a target strategy set satisfying a preset convergence condition.

[0011] In a second aspect, the disclosure provides an intelligent decision-making system based on a distributed training framework, the distributed training framework comprising a sampler, a learner and a coordination server, and the sampler and the learner being multiple;

[0012] The intelligent decision-making system comprises:

[0013] The policy selection module is configured to select k strategies from the first strategy set as training strategies, and determine a sampler and a learner corresponding to each training strategy; the first strategy set is a strategy set initialized in a game environment;

[0014] The sampling module is configured to perform adversarial sampling on each training strategy and a target strategy as a group to obtain first sampling data, the target strategy being a strategy in the first strategy set, and the target strategy being different from the k training strategies;

[0015] The aggregation module is configured to train the learners based on the first sampling data to obtain network model parameters of the learners, and aggregate the network model parameters to obtain global model parameters;

[0016] The iterative training module is configured to send the global model parameters to each learner and each sampler for iterative training to obtain a target strategy set satisfying a preset convergence condition.

[0017] In a third aspect, the disclosure provides an electronic device comprising a memory, a processor and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of the intelligent decision-making method based on the distributed training framework as described above when executing the computer program.

[0018] In a fourth aspect, the disclosure provides a computer-readable storage medium storing a computer program, wherein the computer program is executed by a processor to implement the steps of the intelligent decision-making method based on the distributed training framework as described above.

[0019] The intelligent decision-making method and system based on the distributed training framework provided by the embodiments of the disclosure have the following advantages:

[0020] The parallel strategy sampling and training in multiple simulation interaction environment instances is performed by using the distributed training framework, the synchronous update of the strategy is realized through the distributed architecture, the effective use of data of each strategy in the complete game process is ensured, and the win rate evaluation between two strategies can be completed while the data is collected. The intelligent decision method provided by the present disclosure effectively promotes the diversity of the strategy set, and can significantly improve the strategy diversity optimization efficiency in the military confrontation game scene. The improvement of the diversity helps to find more different and better strategy solutions in the game environment. In addition, the parallel sampler and learner are used, each training strategy can interact with other strategies and collect sample data in an independent environment instance. This parallel processing method significantly improves the training efficiency, so that a large amount of data for strategy optimization can be collected in a short time. At the same time, the distributed training framework can effectively reduce the diversity calculation overhead, and support the strategy optimization demand of large-scale multi-agent system. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained by those skilled in the art without creative labor.

[0022] Figure 1 The flowchart of the intelligent decision method based on the distributed training framework provided by an embodiment of the present disclosure is shown.

[0023] Figure 2 The partial structure diagram of the distributed training framework provided by an embodiment of the present disclosure is shown.

[0024] Figure 3 The structure block diagram of the intelligent decision system based on the distributed training framework provided by an embodiment of the present disclosure is shown.

[0025] Figure 4 The schematic block diagram of the electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0026] In the following description, specific details such as specific system structures, techniques, etc. are presented in order to thoroughly understand the embodiments of the present disclosure. However, it should be clear to those skilled in the art that the present disclosure can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits and methods are omitted to avoid unnecessary details that hinder the description of the present disclosure.

[0027] In order to make the purposes, technical solutions and advantages of the present disclosure clearer, specific embodiments will be described below with reference to the drawings.

[0028] Please refer to Figures 1-2 , Figure 1 The flowchart of the intelligent decision-making method based on the distributed training framework provided by an embodiment of the present disclosure is shown in Figure 2 The partial structure diagram of the distributed training framework provided by an embodiment of the present disclosure is shown in the figure. The intelligent decision-making method comprises the following steps:

[0029] S101: Select k strategies from the first strategy set as training strategies, and determine the sampler and the learner corresponding to each training strategy; the first strategy set is the strategy set initialized in the game environment.

[0030] In the present embodiment, the training environment needs to be initialized. First, the first strategy set is initialized in the game environment. The first strategy set is , wherein each training strategy can be a behavior strategy in the current game environment. The first strategy set can be obtained by prior knowledge or randomly generated.

[0031] Secondly, based on the existing first strategy set Π, k strategies are selected as training strategies for adversarial sampling. The specific selection method is as follows:

[0032] k strategies are selected from the first strategy set Π. (k<n) and the strategy being trained (an arbitrary strategy in the first strategy set) are adversarially sampled, and data samples are collected for policy optimization. The selection of training strategies can be realized according to different requirements, such as randomly selecting k strategies as training strategies or selecting the top k strategies as training strategies according to Nash equilibrium.

[0033] Initialize a learner for each training strategy, i.e. a total of k learners need to be initialized, forming a learner set . A set of parallel samplers is constructed for each training strategy, so that the current training strategy interacts with other strategies in independent environment instances and collects sample data.

[0034] The network model parameters of the learner can be randomly initialized, for example, a specific distribution (such as Gaussian distribution, uniform distribution, etc.) is selected for random initialization to ensure the stability and convergence of the model in the training process. The network model parameters of the learner can also be selectively inherited from the model pool. For example, selective inheritance is performed by fine-tuning, ensemble learning, model selection, etc.

[0035] S102: Adversarial sampling is performed on each training strategy and a target strategy as a group to obtain first sampling data, the target strategy being a strategy in the first strategy set and different from the k training strategies.

[0036] In this embodiment, the construction of the parallel distributed simulation interactive environment needs to ensure that each adversarial strategy has the same number of simulation environments to ensure the fairness of the expected data. Each training strategy that needs to be adversarially sampled can correspond to m simulation interactive environments, and a total of mk parallel simulation interactive environments are needed. The network model parameters of the k learners are initially consistent, and only different learners use different sampling data for training during training, such as the learner uses the target strategy to generate data for training. The target strategy is a strategy in the first strategy set and different from the k training strategies.

[0037] S103: Training the learners based on the first sampling data to obtain network model parameters of the learners, and aggregating the network model parameters to obtain global model parameters.

[0038] In this embodiment, referring to Figure 2 , the kth training strategy has m samplers, and the first sampling data is the data collected by each sampler. The first sampling data includes a first part of sampling data and a second part of sampling data. The first part of sampling data is sent to the sample pool of the kth training strategy corresponding learner for training, and the second part of sampling data includes the expected win rate of the target strategy and the training strategy, and the expected win rate data is sent to the coordination server for data processing. The coordination server calculates the overall diversity score based on the expected win rate data sent by the m samplers, and sends the overall diversity score to the learners for training. Each learner trains based on the corresponding data in the first part of sampling data and the overall diversity score to obtain its own network model training parameter. Each learner sends its own network model training parameter to the coordination server, and the coordination server aggregates the plurality of network model parameters to obtain global model parameters. Figure 2 ①; the second part of sampling data includes the expected win rate of the target strategy and the training strategy, and the expected win rate data is sent to the coordination server for data processing. The coordination server calculates the overall diversity score based on the expected win rate data sent by the m samplers, and sends the overall diversity score to the learners for training. Each learner trains based on the corresponding data in the first part of sampling data and the overall diversity score to obtain its own network model training parameter. Each learner sends its own network model training parameter to the coordination server, and the coordination server aggregates the plurality of network model parameters to obtain global model parameters. Figure 2

[0039] S104: Sending the global model parameters to each learner and each sampler for iterative training to obtain a target strategy set that meets a preset convergence condition.

[0040] In this embodiment, the coordination server synchronizes the aggregated model training parameters to all samplers corresponding to the target strategy and all learners, as shown in Figure 2 ⑤ at this time, all samplers will use the same target strategy​​ The different training strategies are interactively sampled to ensure consistency of the strategies, and each learner also starts training with the same model training parameters in the next round of iteration. This process is repeated until the strategy set reaches the preset convergence condition, thereby achieving the optimization of strategy diversity and performance. The target strategy set is the first strategy set The set composed of the k strategies.

[0041] As can be seen from the above, the present disclosure utilizes a distributed training framework to perform parallel strategy sampling and training in multiple simulated interactive environment instances, and realizes synchronous updating of strategies through a distributed architecture, ensuring effective use of data by each strategy in the complete game process and enabling the win rate evaluation between two strategies to be completed while collecting data. The intelligent decision-making method provided by the present disclosure effectively promotes the diversity of the strategy set and can significantly improve the efficiency of strategy diversity optimization in a military confrontation game scenario. The improvement of diversity helps to find more different and better strategy solutions in the game environment. In addition, the present disclosure uses parallel samplers and learners, and each training strategy can interact with other strategies and collect sample data in an independent environment instance. This parallel processing method significantly improves training efficiency, enabling a large amount of data for strategy optimization to be collected in a short period of time. At the same time, the distributed training framework can effectively reduce the diversity calculation overhead and support the strategy optimization needs of large-scale multi-agent systems.

[0042] In an embodiment of the present disclosure, the first sampling data includes a first part of sampling data and a second part of sampling data;

[0043] Training the learner based on the first sampling data to obtain network model parameters of the learner includes:

[0044] The sampler sends the corresponding first part of sampling data to the sample pool of the learner and sends the corresponding second part of sampling data to the coordination server;

[0045] The coordination server calculates an overall diversity score based on the k sets of second part of sampling data and sends the overall diversity score to the sample pool of each learner, and each learner trains to obtain network model parameters based on the data in the sample pool.

[0046] The second part of sampling data includes the expected win rate of the training strategy and the target strategy;

[0047] The coordination server calculates an overall diversity score based on the k sets of second part of sampling data, including:

[0048] The coordination server calculates the overall diversity score based on the k sets of expected win rates and the first formula, and the first formula is:

[0049]

[0050] wherein, represents the overall diversity score, k represents the k groups of data, represents the target strategy and the expected win rate of the jth training strategy. .

[0051] In this embodiment, is a diversity calculation function. The expected win rate is a statistical quantity representing the average win rate of the training strategy relative to the target strategy in multiple confrontations. This data is crucial for evaluating the performance and diversity of the training strategy.

[0052] The training process of the learner is as follows:

[0053] Data acquisition: the sampler sends the first part of the sampled data to the sample pool of the learner for the learner to train. The coordination server calculates the overall diversity score based on the k groups of the second part of the sampled data (the expected win rates of the k training strategies and the target strategy), and sends the overall diversity score to the learner for training.

[0054] Model training: after receiving the overall diversity score sent by the coordination server, each learner combines it with the first part of the sampled data in the sample pool to update the network model parameters. In this way, the learner not only optimizes the performance of the strategy, but also considers the diversity of the strategy.

[0055] From the above, it can be concluded that in the game environment, the diversity of the strategy is crucial for finding the optimal solution. By calculating the overall diversity score, the learner can be guided to not only focus on the performance of the strategy (i.e. win rate) during the training process, but also focus on the difference and diversity between strategies. This helps to avoid the strategy falling into a local optimal solution, and improves the robustness and generalization ability of the strategy set.

[0056] In an embodiment of the present disclosure, each learner obtains network model parameters by training according to the data in the sample pool, comprising:

[0057] Each learner updates the original reward function to obtain a target reward function according to the data in the corresponding sample pool, and each learner trains to obtain network model parameters according to the corresponding target reward function.

[0058] In this embodiment, each learner obtains data from its own sample pool. The data in these sample pools is collected through interaction with the environment, confrontation with other strategies, etc., and contains key information such as state, action, and reward. The learner does not rely solely on the original reward function. Each learner has a corresponding reward function , such as Figure 2③. After receiving the overall diversity score sent by the coordination server, the learner updates the original reward function based on the data in the sample pool and the overall diversity score to obtain a new target reward function.

[0059] The learner starts training the network model parameters using the updated target reward function, aiming to find a set of network model parameters that enable the learner to select the optimal action in a given state, thereby maximizing the expected value of the target reward function.

[0060] From the above, it can be concluded that by introducing the overall diversity score to update the original reward function, each learner not only focuses on the performance improvement of a single strategy, but also pays attention to the diversity and complementarity between strategies. This mechanism encourages learners to explore a more diverse policy space during training, reducing the risk of falling into a local optimal solution. At the same time, using the updated target reward function for training, the learner can more effectively adapt to complex and variable environments, improving the robustness and generalization ability of the strategy.

[0061] In an embodiment of the present disclosure, each learner updates the original reward function based on the data in the corresponding sample pool to obtain a target reward function, comprising:

[0062] Each learner updates the original reward function based on the data in the corresponding sample pool and the second formula to obtain a target reward function, and the second formula is:

[0063]

[0064] wherein, represents the target strategy is the jth training strategy is the original reward function obtained by confrontation, represents the target reward function, is the network model weight coefficient of the learner, represents the diversity score.

[0065] In an embodiment of the present disclosure, the network model parameters are aggregated to obtain global model parameters, comprising:

[0066] The global model parameters are calculated according to the third formula, and the third formula is:

[0067]

[0068] wherein, represents the global model parameters, represents the network model parameters of the jth learner, and k represents the number of learners.

[0069] In this embodiment, the learners independently train using the data in their respective sample pools, and the neural network model parameters of each learner are usually inconsistent after completing this round of training. The learners train based on the updated reward function . The trained network model parameters are sent to the coordination server, as shown in Figure 2 . After receiving the updated network model parameters of the learners , the coordination server aggregates these network model parameters to obtain new global model parameters .

[0070] From the above, it can be concluded that by introducing the diversity score to adjust the reward function, the embodiment promotes the strategy diversity among the learners and avoids the problem of strategy homogeneity in the training process. At the same time, the coordination server aggregates the network model parameters trained by the learners to obtain more comprehensive and robust global model parameters. This distributed training and parameter aggregation method not only improves the training efficiency, but also enhances the generalization ability and adaptability of the model, providing strong support for the processing of complex tasks.

[0071] The intelligent decision-making method based on the distributed training framework corresponding to the above embodiment, Figure 3 the structure diagram of the intelligent decision-making system based on the distributed training framework provided by an embodiment of the present disclosure. For ease of illustration, only parts related to the embodiments of the present disclosure are shown. Referring to Figure 3 , the intelligent decision-making system based on the distributed training framework 20 includes a strategy selection module 21, a sampling module 22, an aggregation module 23, and an iterative training module 24.

[0072] The strategy selection module 21 is configured to select k strategies from a first strategy set as training strategies, and determine a sampler and a learner corresponding to each training strategy; the first strategy set is a strategy set initialized in a game environment;

[0073] The sampling module 22 is configured to perform adversarial sampling on each training strategy and a target strategy as a group to obtain first sampling data, the target strategy being a strategy in the first strategy set and different from the k training strategies;

[0074] The aggregation module 23 is configured to train the learners based on the first sampling data to obtain network model parameters of the learners, and aggregate the network model parameters to obtain global model parameters;

[0075] The iterative training module 24 is configured to send the global model parameters to each learner and each sampler for iterative training to obtain a target strategy set satisfying a preset convergence condition.

[0076] In an embodiment of the present disclosure, the first sampling data includes a first part of sampling data and a second part of sampling data. The aggregation module 23 is specifically configured to:

[0077] The sampler sends the corresponding first part of sampling data to the sample pool of the learner, and sends the corresponding second part of sampling data to the coordination server;

[0078] The coordination server calculates an overall diversity score based on the k groups of second part of sampling data, and sends the overall diversity score to the sample pool of each learner, and each learner trains to obtain network model parameters according to the data in the sample pool.

[0079] In an embodiment of the present disclosure, the second part of sampling data includes an expected win rate of the training strategy and the target strategy. The aggregation module 23 is specifically configured to:

[0080] The coordination server calculates an overall diversity score based on the k groups of expected win rates and a first formula, and the first formula is:

[0081]

[0082] wherein, represents the overall diversity score, k represents the k groups of data, represents the target strategy and the expected win rate of the jth training strategy .

[0083] In an embodiment of the present disclosure, the aggregation module 23 is specifically configured to:

[0084] Each learner updates the original reward function to obtain a target reward function according to the data in the corresponding sample pool, and each learner trains to obtain network model parameters according to the corresponding target reward function.

[0085] In an embodiment of the present disclosure, the aggregation module 23 is specifically configured to:

[0086] Each learner updates the original reward function to obtain a target reward function according to the data in the corresponding sample pool and a second formula, and the second formula is:

[0087]

[0088] wherein, represents the target strategy and the original reward function obtained by the jth training strategy , represents the target reward function, is a network model weight coefficient of the learner, represents the diversity score.

[0089] In an embodiment of the present disclosure, the aggregation module 23 is specifically configured to:

[0090] The global model parameters are calculated according to a third formula, which is:

[0091]

[0092] wherein, represents the global model parameters, represents the network model parameters of the jth learner, and k represents the number of learners.

[0093] In an embodiment of the present disclosure, each training strategy corresponds to a set of parallel distributed simulation interaction environments, one or more samplers, and one learner.

[0094] In an embodiment of the present disclosure, referring to Figure 2 , m represents the number of samplers, is the target strategy, i.e., the strategy currently running, is the kth training strategy, i.e., the opponent strategy. is the expected win rate of the target strategy and the opponent strategy, represents the network model parameters of the learner .

[0095] Referring to Figure 4 , Figure 4 is a schematic block diagram of an electronic device provided by an embodiment of the present disclosure. As shown in the electronic device 300 in the embodiment, the electronic device 300 can include one or more processors 301, one or more input devices 302, one or more output devices 303, and one or more memories 304. The above-mentioned processor 301, input device 302, output device 303, and memory 304 complete mutual communication through a communication bus 305. The memory 304 is used to store a computer program, and the computer program includes program instructions. The processor 301 is used to execute the program instructions stored in the memory 304. Wherein, the processor 301 is configured to call the program instructions to execute the functions of each module / unit in the above-mentioned device embodiments, such as the functions of the modules 21 to 24 shown in Figure 4 Figure 3

[0096] ​​It should be appreciated that in the embodiments of the present disclosure, the processor 301 can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0097] The input device 302 can include a touchpad, a fingerprint collection sensor (for collecting fingerprint information and direction information of a fingerprint of a user), a microphone, etc., and the output device 303 can include a display (LCD, etc.), a speaker, etc.

[0098] The memory 304 can include a read-only memory and a random access memory, and provide instructions and data for the processor 301. A portion of the memory 304 can also include a non-volatile random access memory. For example, the memory 304 can also store device type information.

[0099] In specific implementations, the processor 301, the input device 302, and the output device 303 described in the embodiments of the present disclosure can execute the implementation manners described in the first and second embodiments of the intelligent decision-making method based on a distributed training framework provided by the embodiments of the present disclosure, and can also execute the implementation manners of the electronic device described in the embodiments of the present disclosure, which will not be described here again.

[0100] In another embodiment of the present disclosure, a computer readable storage medium is provided, which stores a computer program. The computer program includes program instructions, which, when executed by a processor, implement all or part of the processes of the above-mentioned embodiment methods. The computer program can also instruct related hardware to complete the above-mentioned processes. The computer program can be stored in a computer readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate form. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0101] The computer readable storage medium can be an internal storage unit of the electronic device, such as a hard disk or a memory of the electronic device. The computer readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the computer readable storage medium can include both the internal storage unit and the external storage device of the electronic device. The computer readable storage medium is used to store the computer program and other programs and data required by the electronic device. The computer readable storage medium can also be used to temporarily store data that has been output or will be output.

[0102] Those skilled in the art can appreciate that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in general terms in the above description. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present disclosure.

[0103] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the electronic device and the units described above can refer to the corresponding processes in the above-mentioned method embodiments, which will not be described here.

[0104] In several embodiments provided in the present application, it should be understood that the disclosed electronic device and method can be implemented in other manners. For example, the embodiments of the apparatus described above are merely illustrative. For example, the division of the units is only a logical function division. There can be another division manner for the actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed coupling or direct coupling or communication connection between the units can be indirect coupling or communication connection through some interfaces, or can be in electrical, mechanical or other forms.

[0105] The units described as separated components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purposes of the embodiments of the present disclosure.

[0106] In addition, each functional unit in the various embodiments of the present disclosure can be integrated in one processing unit, or each unit can exist physically as a separate unit, or two or more units can be integrated in one unit. The integrated unit can be implemented in the form of hardware, or in the form of a software functional unit.

[0107] The above is merely specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto, and any skilled person in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present disclosure, and these modifications or replacements should be covered in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. An intelligent decision-making method based on a distributed training framework, characterized in that, The distributed training framework comprises a sampler, a learner and a coordination server, and the sampler and the learner are multiple; The intelligent decision-making method comprises: selecting k strategies from a first strategy set as training strategies, and determining a sampler and a learner corresponding to each training strategy; the first strategy set is a strategy set initialized in a game environment; performing adversarial sampling on each training strategy and a target strategy as a group to obtain first sampling data, the target strategy being a strategy in the first strategy set and different from the k training strategies; each training strategy corresponds to a group of parallel distributed simulation interactive environments, one or more samplers and one learner; training the learners based on the first sampling data to obtain network model parameters of the learners, and aggregating the network model parameters to obtain global model parameters; sending the global model parameters to each learner and each sampler for iterative training to obtain a target strategy set satisfying a preset convergence condition; The first sampling data comprises first part sampling data and second part sampling data, the kth training strategy has m samplers, the first sampling data is data collected by each sampler, and the second part sampling data comprises an expected win rate of the target strategy and the training strategies; The training of the learners based on the first sampling data to obtain network model parameters of the learners comprises: for the kth learner: sending the first part sampling data to a sample pool of the learner corresponding to the kth training strategy for training; and sending the expected win rate in the second part sampling data to the coordination server for data processing, so that the coordination server calculates an overall diversity score based on the expected win rates sent by the m samplers, and sends the overall diversity score to the sample pool of the learner for training; each learner trains based on k sets of overall diversity scores in the sample pool to obtain respective network model training parameters; the k sets of overall diversity scores are calculated by the coordination server based on the expected win rates in the k sets of second part sampling data; The aggregation of the network model parameters to obtain global model parameters comprises: The k learners in the distributed training framework send respective network model parameters to the coordination server, and the coordination server aggregates the k network model parameters to obtain global model parameters.

2. The intelligent decision-making method based on the distributed training framework according to claim 1, wherein: calculating k sets of overall diversity scores based on the expected win rates in the k sets of second part sampling data comprises: calculating the overall diversity scores based on the k sets of expected win rates and a first formula, the first formula being: wherein, represents the kth group overall diversity score, k represents the kth group of data, represents the target strategy and the jth training strategy expected win rate. 3.The intelligent decision-making method based on a distributed training framework of claim 1, wherein, The training of each learner based on the k sets of overall diversity scores in the sample pool to obtain respective network model training parameters comprises: each learner updates an original reward function based on data in the corresponding sample pool to obtain a target reward function, and each learner trains based on the respective target reward function to obtain network model parameters. 4.The intelligent decision-making method based on a distributed training framework of claim 3, wherein, The updating of the original reward function by each learner based on data in the corresponding sample pool to obtain a target reward function comprises: Each learner updates the original reward function according to data in the corresponding sample pool and a second formula to obtain a target reward function, the second formula being: wherein, represents a target policy with the jth training policy against the resulting raw reward function, represents a target reward function, is a network model weight coefficient of the learner, represents k sets of overall diversity scores. 5.The intelligent decision-making method based on a distributed training framework of claim 1, wherein, The aggregating of the k network model parameters to obtain the global model parameter comprises: The global model parameter is calculated according to a third formula, the third formula being: wherein, denotes global model parameters, denotes network model parameters of the jth learner, k denotes the number of learners, and also denotes k groups of data.

6. An intelligent decision system based on a distributed training framework, characterized in that, The distributed training framework comprises a sampler, a learner and a coordination server, and the sampler and the learner are multiple; The intelligent decision-making system comprises: The policy selection module is configured to select k strategies from a first strategy set as training strategies, and determine a sampler and a learner corresponding to each training strategy; the first strategy set is a strategy set initialized in the game environment; The sampling module is configured to perform adversarial sampling on each training strategy and a target strategy as a group to obtain first sampling data, the target strategy being a strategy in the first strategy set and different from the k training strategies; each training strategy corresponds to a group of parallel distributed simulation interactive environments, one or more samplers and one learner; the first sampling data comprises a first part of sampling data and a second part of sampling data, the kth training strategy having m samplers, the first sampling data being data collected by each sampler, and the second part of sampling data comprising an expected win rate of the target strategy and the training strategies; The aggregation module is configured to train the learners based on the first sampling data to obtain network model parameters of the learners, and aggregate the network model parameters to obtain a global model parameter; The iterative training module is configured to send the global model parameter to each learner and each sampler for iterative training to obtain a target strategy set satisfying a preset convergence condition; The aggregation module is specifically configured to: send the first part of sampling data to a sample pool of the kth training strategy corresponding learner for training; and send the expected win rate in the second part of sampling data to the coordination server for data processing, so that the coordination server calculates an overall diversity score based on the expected win rates sent by the m samplers, and sends the overall diversity score to the sample pool of the learner for training; Each learner trains based on k groups of overall diversity scores in the sample pool to obtain respective network model training parameters; the k groups of overall diversity scores are calculated by the coordination server based on the expected win rates in the k groups of second part of sampling data; The aggregation module is specifically configured to: the k learners in the distributed training framework send respective network model parameters to the coordination server, and the coordination server aggregates the k network model parameters to obtain the global model parameter.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, The processor executes the computer program to implement the steps of the method of any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, the computer-readable storage medium comprising: The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 5.

Images