Intelligent decision-making method and system based on distributed training framework

Abstract

The invention provides an intelligent decision-making method and system based on a distributed training framework, and belongs to the field of artificial intelligence and distributed computing, and the distributed training framework comprises a sampler, a learning device and a coordination server. The intelligent decision-making method comprises the following steps: selecting k strategies from a first strategy set as training strategies, and determining a sampler and a learning device corresponding to each training strategy; and taking each training strategy and the target strategy as a group for adversarial sampling to obtain first sampling data. And training the learner based on the first sampling data to obtain network model parameters of the learner, and aggregating the network model parameters to obtain global model parameters. And sending the global model parameters to each learner and each sampler for iterative training to obtain a target strategy set meeting a preset convergence condition. According to the method, synchronous sampling and distributed parallel computing are combined, so that intelligent decision is suitable for a complex military confrontation game scene, and the optimization effect of strategy diversity is remarkably improved.

Description

Technical Field

[0001] The present disclosure belongs to the field of artificial intelligence and distributed computing technology, and more specifically, to an intelligent decision-making method and system based on a distributed training framework. Background Art

[0002] Military confrontation games can usually be modeled as two-person zero-sum games. In related research, diversity optimization is a key means to improve the robustness and adaptability of intelligent agent strategies. In complex military confrontation game scenarios, existing diversity optimization methods are usually achieved by adjusting reward functions, encouraging exploration behaviors, and other means, mainly based on the event level composed of state features or state trajectories. However, these methods are limited to adjustments at the state and event levels, and have limited effects on the diversity optimization of the overall strategy. Therefore, we need to consider diversity optimization from a more comprehensive strategic level.

[0003] In simple matrix games, some theoretical studies have attempted to optimize diversity from the strategy level, called response diversity. Response diversity helps agents avoid being predicted and exploited by opponents by adjusting the interactions between strategies. However, this approach is mainly applied to simple matrix games, and agents cannot make effective decisions in complex game scenarios. Summary of the invention

[0004] The purpose of the present disclosure is to provide an intelligent decision-making method and system based on a distributed training framework, so that intelligent decision-making can be applied to complex military confrontation game scenarios and significantly improve the optimization effect of strategy diversity.

[0005] In a first aspect of an embodiment of the present disclosure, an intelligent decision-making method based on a distributed training framework is provided, wherein the distributed training framework includes a sampler, a learner, and a coordination server, and there are multiple samplers and learners; The intelligent decision-making method comprises: Select k strategies from the first strategy set as training strategies, and determine the sampler and learner corresponding to each training strategy; the first strategy set is the strategy set initialized in the game environment; Perform adversarial sampling on each training strategy and the target strategy as a group to obtain first sampling data, wherein the target strategy is a strategy in the first strategy set, and the target strategy is different from the k training strategies; Based on the first sampling data, the learner is trained to obtain network model parameters of the learner, and the network model parameters are aggregated to obtain global model parameters; The global model parameters are sent to each learner and each sampler for iterative training to obtain a target strategy set that meets a preset convergence condition.

[0006] A second aspect of the embodiments of the present disclosure provides an intelligent decision-making system based on a distributed training framework, wherein the distributed training framework includes a sampler, a learner, and a coordination server, and there are multiple samplers and learners; The intelligent decision-making system comprises: A strategy selection module, used to select k strategies from a first strategy set as training strategies, and determine a sampler and a learner corresponding to each training strategy; the first strategy set is a strategy set initialized in a game environment; A sampling module, used for performing adversarial sampling on each training strategy and a target strategy as a group to obtain first sampling data, wherein the target strategy is a strategy in the first strategy set, and the target strategy is different from the k training strategies; an aggregation module, configured to train a learner based on the first sampling data to obtain network model parameters of the learner, and aggregate the network model parameters to obtain global model parameters; The iterative training module is used to send the global model parameters to each learner and each sampler for iterative training to obtain a target strategy set that meets a preset convergence condition.

[0007] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of the above-mentioned intelligent decision-making method based on a distributed training framework when executing the computer program.

[0008] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned intelligent decision-making method based on a distributed training framework are implemented.

[0009] The beneficial effects of the intelligent decision-making method and system based on the distributed training framework provided by the embodiments of the present disclosure are: The present disclosure utilizes a distributed training framework to perform parallel strategy sampling and training in multiple simulation interactive environment instances, realizes synchronous update of strategies through a distributed architecture, ensures effective use of data for each strategy in the complete game process, and can complete the evaluation of the winning rate between two strategies while collecting data. The intelligent decision-making method provided by the present disclosure effectively promotes the diversity of strategy sets and can significantly improve the optimization efficiency of strategy diversity in military confrontation game scenarios. The improvement of diversity helps to find more different and better strategy solutions in the game environment. In addition, the present disclosure uses parallel samplers and learners, and each training strategy can interact with other strategies in independent environment instances and collect sample data. This parallel processing method significantly improves the training efficiency, so that a large amount of data for strategy optimization can be collected in a short time. At the same time, the distributed training framework can effectively reduce the diversity computing overhead and support the strategy optimization requirements of large-scale multi-agent systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0011] Figure 1 A flowchart of an intelligent decision-making method based on a distributed training framework provided in one embodiment of the present disclosure; Figure 2 A partial structural diagram of a distributed training framework provided in one embodiment of the present disclosure; Figure 3 A structural block diagram of an intelligent decision-making system based on a distributed training framework provided in one embodiment of the present disclosure; Figure 4 A schematic block diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0012] In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present disclosure. However, it should be clear to those skilled in the art that the present disclosure may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present disclosure with unnecessary details.

[0013] In order to make the purpose, technical solutions and advantages of the present disclosure more clear, specific embodiments will be described below in conjunction with the accompanying drawings.

[0014] Please refer to Figure 1-Figure 2 , Figure 1 which is a schematic flowchart of an intelligent decision-making method based on a distributed training framework provided by an embodiment of the present disclosure. Figure 2 which is a partial structural schematic diagram of a distributed training framework provided by an embodiment of the present disclosure. The intelligent decision-making method includes: S101: Select k strategies from the first strategy set as training strategies, and determine the sampler and learner corresponding to each training strategy; the first strategy set is the strategy set initialized in the game environment.

[0015] In this embodiment, it is necessary to initialize the training environment. First, initialize the first strategy set in the game environment. The first strategy set is , where each training strategy can be a behavioral strategy in the current game environment. The first strategy set can be obtained through prior knowledge or random generation.

[0016] Secondly, based on the existing first strategy set Π, select k strategies as training strategies for adversarial sampling. The specific selection method is as follows: Select k strategies from the first strategy set (k < n) and the strategy being trained (any strategy in the first strategy set) for adversarial sampling to collect data samples for policy optimization. The selection of training strategies can be implemented according to different requirements, such as randomly selecting k strategies as training strategies or selecting the top k best strategies according to the Nash equilibrium as training strategies.

[0017] Initialize a learner for each training strategy, that is, a total of k learners need to be initialized to form a learner set . Construct a group of parallel samplers for each training strategy, so that the current training strategy interacts with other strategies in an independent environment instance and collects sample data.

[0018] The network model parameters of the learner can be randomly initialized. For example, select a specific distribution (such as Gaussian distribution, uniform distribution, etc.) for random initialization to ensure the stability and convergence of the model during training. The network model parameters of the learner can also be selectively inherited from the model pool. For example, through fine-tuning, ensemble learning, model selection, etc. for selective inheritance.

[0019] S102: Take each training strategy and the target strategy as a group for adversarial sampling to obtain the first sampling data. The target strategy is a strategy in the first strategy set, and the target strategy is different from the k training strategies.

[0020] In this embodiment, the construction of a parallel distributed simulation interaction environment ensures that each adversarial strategy has the same number of simulation environments to ensure the fairness of the expected data. Each training strategy that requires adversarial sampling can correspond to m simulation interaction environments, so a total of mk parallel simulation interaction environments are required. The initial network model parameters of the k learners are completely consistent, but different learners use different sampling data for training. For example, Using Targeted Strategies With the first training strategy The target strategy is the strategy in the first strategy set, and the target strategy is different from the k training strategies.

[0021] S103: Training the learner based on the first sampling data to obtain network model parameters of the learner, and aggregating the network model parameters to obtain global model parameters.

[0022] In this embodiment, reference Figure 2 , the kth training strategy has m samplers, and the first sampling data is the data collected by each sampler. The first sampling data includes the first part of sampling data and the second part of sampling data. Among them, the first part of sampling data will be sent to the sample pool of the learner corresponding to the kth training strategy for training, refer to Figure 2 ①; The second part of the sampling data includes the expected winning rate of the target strategy and the training strategy, refer to Figure 2 ②, these expected win rate data will be sent to the coordination server for data processing. The coordination server calculates the overall diversity score based on the expected win rate data sent by the m samplers, and sends the overall diversity score to the learner for training. Each learner is trained based on the corresponding data in the first part of the sampled data and the overall diversity score to obtain its own network model training parameters. Each learner sends its own network model training parameters to the coordination server, and the coordination server aggregates multiple network model parameters to obtain global model parameters.

[0023] S104: Sending the global model parameters to each learner and each sampler for iterative training to obtain a target strategy set that meets a preset convergence condition.

[0024] In this embodiment, the coordination server synchronizes the aggregated model training parameters to the target strategies corresponding to all samplers. And all learners, such as Figure 2 As shown in ⑤, all samplers will use the same target strategy at this time Interactive sampling of different training strategies ensures the consistency of strategies. At the same time, each learner will start training with the same model training parameters in the next iteration. This process is repeated until the strategy set reaches the preset convergence condition, thus achieving the best strategy diversity and performance. The target strategy set is the first strategy set. is a set of k strategies.

[0025] It can be concluded from the above that the present disclosure uses a distributed training framework to perform parallel strategy sampling and training in multiple simulation interactive environment instances, and realizes synchronous update of strategies through a distributed architecture, ensuring the effective use of data for each strategy in the complete game process, and can complete the winning rate evaluation between two strategies while collecting data. The intelligent decision-making method provided by the present disclosure effectively promotes the diversity of strategy sets and can significantly improve the optimization efficiency of strategy diversity in military confrontation game scenarios. The improvement of diversity helps to find more different and better strategy solutions in the game environment. In addition, the present disclosure uses parallel samplers and learners, and each training strategy can interact with other strategies in independent environment instances and collect sample data. This parallel processing method significantly improves the training efficiency, so that a large amount of data for strategy optimization can be collected in a short time. At the same time, the distributed training framework can effectively reduce the diversity computing overhead and support the strategy optimization requirements of large-scale multi-agent systems.

[0026] In one embodiment of the present disclosure, the first sampled data includes a first portion of sampled data and a second portion of sampled data; The learner is trained based on the first sampled data to obtain the network model parameters of the learner, including: The sampler sends the corresponding first part of the sampled data to the sample pool of the learner, and sends the corresponding second part of the sampled data to the coordination server; The coordination server calculates the overall diversity score based on the k groups of second part sampling data, and sends the overall diversity score to the sample pool of each learner. Each learner is trained according to the data in the sample pool to obtain the network model parameters.

[0027] The second part of the sampling data includes the expected winning rate of the training strategy and the target strategy; The coordination server calculates the overall diversity score based on the k sets of second part sampled data, including: The coordination server calculates the overall diversity score based on the expected win rate of k groups and the first formula, which is:

[0028] in, represents the overall diversity score, k represents k groups of data, Indicates the target strategy With the jth training strategy Expected win rate.

[0029] In this embodiment, is a diversity calculation function. The expected winning rate is a statistic that represents the average winning rate of the training strategy relative to the target strategy in multiple confrontations. This data is crucial for evaluating the performance and diversity of the training strategy.

[0030] The training process of the learner is: Data acquisition: The sampler sends the first part of the sampled data to the sample pool of the learner for training. The coordination server calculates the overall diversity score based on the k groups of the second part of the sampled data (the expected winning rate of the k training strategies and the target strategy), and sends the overall diversity score to the learner for training.

[0031] Model training: After receiving the overall diversity score sent by the coordination server, each learner combines it with the first part of the sampled data in the sample pool to update the network model parameters. In this way, the learner will consider the diversity of the strategy while optimizing the performance of the strategy.

[0032] From the above, we can conclude that in a game environment, the diversity of strategies is crucial to finding the optimal solution. By calculating the overall diversity score, we can guide the learner to focus not only on the performance of the strategy (i.e., the winning rate) during the training process, but also on the differences and diversity between strategies. This helps to avoid the strategy from falling into the local optimal solution and improves the robustness and generalization ability of the strategy set.

[0033] In one embodiment of the present disclosure, each learner is trained according to the data in the sample pool to obtain network model parameters, including: Each learner updates the original reward function according to the data in the corresponding sample pool to obtain the target reward function, and each learner is trained according to its corresponding target reward function to obtain the network model parameters.

[0034] In this embodiment, each learner obtains data from its own sample pool. The data in these sample pools are collected by interacting with the environment and competing with other strategies, and contain key information such as state, action, and reward. The learner does not only rely on the original reward function. Each learner has a corresponding reward function. ,like Figure 2 As shown in ③. After receiving the overall diversity score sent by the coordination server, the learner will update the original reward function according to the data in the sample pool and the overall diversity score to obtain a new target reward function.

[0035] The learner uses the updated target reward function to start training its network model parameters. The goal is to find a set of network model parameters so that under a given state, the learner can select the optimal action to maximize the expected value of the target reward function.

[0036] It can be concluded from the above that this embodiment updates the original reward function by introducing an overall diversity score. Each learner not only focuses on the performance improvement of a single strategy, but also pays attention to the diversity and complementarity between strategies. This mechanism prompts the learner to explore a more diverse strategy space during the training process, reducing the risk of falling into a local optimal solution. At the same time, by training with the updated target reward function, the learner can more effectively adapt to complex and changing environments, improving the robustness and generalization ability of the strategy.

[0037] In one embodiment of the present disclosure, each learner updates the original reward function according to the data in the corresponding sample pool to obtain the target reward function, including: Each learner updates the original reward function according to the data in the corresponding sample pool and the second formula to obtain the target reward function. The second formula is:

[0038] in, Indicates the target strategy With the jth training strategy The original reward function obtained by adversarial represents the target reward function, is the network model weight coefficient of the learner, Represents the diversity score.

[0039] In one embodiment of the present disclosure, aggregating network model parameters to obtain global model parameters includes: The global model parameters are calculated according to the third formula, which is:

[0040] in, represents the global model parameters, represents the network model parameters of the jth learner, and k represents the number of learners.

[0041] In this embodiment, the learners use the data in their respective sample pools for independent training. Usually, after completing this round of training, the neural network model parameters of each learner are inconsistent. The learners are based on the updated reward function The trained network model parameters are sent to the coordination server, such as Figure 2 ④ As shown in the figure, the coordination server receives the updated network model parameters from the learner. After that, these network model parameters are aggregated to obtain new global model parameters .

[0042] It can be concluded from the above that this embodiment promotes the diversity of strategies among learners and avoids the problem of strategy homogeneity during training by introducing a diversity score adjustment reward function. At the same time, the coordination server aggregates the network model parameters after the learner training to obtain more comprehensive and robust global model parameters. This distributed training and parameter aggregation method not only improves the training efficiency, but also enhances the generalization ability and adaptability of the model, providing strong support for the processing of complex tasks.

[0043] Corresponding to the intelligent decision-making method based on the distributed training framework in the above embodiment, Figure 3 This is a block diagram of the structure of an intelligent decision-making system based on a distributed training framework provided in one embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 3 The intelligent decision-making system 20 based on the distributed training framework includes: a strategy selection module 21, a sampling module 22, an aggregation module 23 and an iterative training module 24.

[0044] The strategy selection module 21 is used to select k strategies from the first strategy set as training strategies and determine the sampler and learner corresponding to each training strategy; the first strategy set is a strategy set initialized in the game environment; A sampling module 22 is used to perform adversarial sampling on each training strategy and a target strategy as a group to obtain first sampling data, where the target strategy is a strategy in the first strategy set, and the target strategy is different from the k training strategies; Aggregation module 23, used for training the learner based on the first sampling data to obtain network model parameters of the learner, and aggregating the network model parameters to obtain global model parameters; The iterative training module 24 is used to send the global model parameters to each learner and each sampler for iterative training to obtain a target strategy set that meets a preset convergence condition.

[0045] In one embodiment of the present disclosure, the first sampled data includes a first portion of sampled data and a second portion of sampled data. The aggregation module 23 is specifically configured to: The sampler sends the corresponding first part of the sampled data to the sample pool of the learner, and sends the corresponding second part of the sampled data to the coordination server; The coordination server calculates the overall diversity score based on the k groups of second part sampling data, and sends the overall diversity score to the sample pool of each learner. Each learner is trained according to the data in the sample pool to obtain the network model parameters.

[0046] In one embodiment of the present disclosure, the second part of the sampled data includes the expected winning rate of the training strategy and the target strategy. The aggregation module 23 is specifically used for: The coordination server calculates the overall diversity score based on the expected win rate of k groups and the first formula, which is:

[0047] in, represents the overall diversity score, k represents k groups of data, Indicates the target strategy With the jth training strategy Expected win rate.

[0048] In one embodiment of the present disclosure, the aggregation module 23 is specifically used to: Each learner updates the original reward function according to the data in the corresponding sample pool to obtain the target reward function, and each learner is trained according to its corresponding target reward function to obtain the network model parameters.

[0049] In one embodiment of the present disclosure, the aggregation module 23 is specifically used to: Each learner updates the original reward function according to the data in the corresponding sample pool and the second formula to obtain the target reward function. The second formula is:

[0050] in, Indicates the target strategy With the jth training strategy The original reward function obtained by adversarial represents the target reward function, is the network model weight coefficient of the learner, Represents the diversity score.

[0051] In one embodiment of the present disclosure, the aggregation module 23 is specifically used to: The global model parameters are calculated according to the third formula, which is:

[0052] in, represents the global model parameters, represents the network model parameters of the jth learner, and k represents the number of learners.

[0053] In one embodiment of the present disclosure, each training strategy corresponds to a set of parallel distributed simulation interaction environments, one or more samplers, and a learner.

[0054] In one embodiment of the present disclosure, reference Figure 2, m represents the number of samplers, is the target policy, that is, the policy currently running. is the kth training strategy, i.e., the opponent strategy. is the expected winning rate of the target strategy and the opponent strategy, Representation Learner The network model parameters.

[0055] See also Figure 4 , Figure 4 A schematic block diagram of an electronic device provided by an embodiment of the present disclosure. Figure 4 The electronic device 300 in the embodiment shown may include: one or more processors 301, one or more input devices 302, one or more output devices 303 and one or more memories 304. The processors 301, input devices 302, output devices 303 and memories 304 communicate with each other via a communication bus 305. The memory 304 is used to store computer programs, which include program instructions. The processor 301 is used to execute the program instructions stored in the memory 304. The processor 301 is configured to call the program instructions to execute the functions of each module / unit in the above-mentioned device embodiments, such as Figure 3 The functions of modules 21 to 24 are shown.

[0056] It should be understood that in the embodiment of the present disclosure, the processor 301 may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0057] The input device 302 may include a touch panel, a fingerprint collection sensor (for collecting the user's fingerprint information and fingerprint direction information), a microphone, etc., and the output device 303 may include a display (LCD, etc.), a speaker, etc.

[0058] The memory 304 may include a read-only memory and a random access memory, and provide instructions and data to the processor 301. A portion of the memory 304 may also include a non-volatile random access memory. For example, the memory 304 may also store information about the device type.

[0059] In a specific implementation, the processor 301, input device 302, and output device 303 described in the embodiments of the present disclosure can execute the implementation methods described in the first and second embodiments of the intelligent decision-making method based on the distributed training framework provided in the embodiments of the present disclosure, and can also execute the implementation methods of the electronic device described in the embodiments of the present disclosure, which will not be repeated here.

[0060] In another embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by the processor, all or part of the processes in the above-mentioned embodiment method are implemented, and the computer program can also be completed by instructing the relevant hardware through the computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, the steps of each of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0061] The computer-readable storage medium may be an internal storage unit of the electronic device of any of the aforementioned embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of the electronic device and an external storage device. The computer-readable storage medium is used to store computer programs and other programs and data required by the electronic device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0062] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0063] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the electronic devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0064] In the several embodiments provided in the present application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces or units, or it can be an electrical, mechanical or other form of connection.

[0065] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present disclosure.

[0066] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0067] The above are only specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present disclosure, and these modifications or replacements should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. An intelligent decision-making method based on a distributed training framework, characterized in that: The distributed training framework includes a sampler, a learner and a coordination server, and there are multiple samplers and learners; The intelligent decision-making method comprises: Select k strategies from the first strategy set as training strategies, and determine the sampler and learner corresponding to each training strategy; the first strategy set is the strategy set initialized in the game environment; Perform adversarial sampling on each training strategy and the target strategy as a group to obtain first sampling data, wherein the target strategy is a strategy in the first strategy set, and the target strategy is different from the k training strategies; Based on the first sampling data, the learner is trained to obtain network model parameters of the learner, and the network model parameters are aggregated to obtain global model parameters; The global model parameters are sent to each learner and each sampler for iterative training to obtain a target strategy set that meets a preset convergence condition.

2. The intelligent decision-making method based on a distributed training framework according to claim 1, characterized in that: The first sampling data includes a first part of sampling data and a second part of sampling data; The step of training the learner based on the first sampled data to obtain the network model parameters of the learner includes: The sampler sends the corresponding first part of the sampled data to the sample pool of the learner, and sends the corresponding second part of the sampled data to the coordination server; The coordination server calculates an overall diversity score based on the k groups of second part sampled data, and sends the overall diversity score to a sample pool of each learner. Each learner is trained according to the data in the sample pool to obtain a network model parameter.

3. The intelligent decision-making method based on a distributed training framework as claimed in claim 2, characterized in that: The second part of the sampled data includes the expected winning rates of the training strategy and the target strategy; The coordination server calculates the overall diversity score based on the k sets of second part sampled data, including: The coordination server calculates the overall diversity score based on the expected win rates of the k groups and the first formula, where the first formula is: in, represents the overall diversity score, k represents k groups of data, Indicates the target strategy With the jth training strategy Expected win rate.

4. The intelligent decision-making method based on a distributed training framework as claimed in claim 2, characterized in that: Each learner is trained according to the data in the sample pool to obtain network model parameters, including: Each learner updates the original reward function according to the data in the corresponding sample pool to obtain the target reward function, and each learner is trained according to its corresponding target reward function to obtain the network model parameters.

5. The intelligent decision-making method based on a distributed training framework as claimed in claim 4, characterized in that: Each learner updates the original reward function according to the data in the corresponding sample pool to obtain the target reward function, including: Each learner updates the original reward function according to the data in the corresponding sample pool and the second formula to obtain the target reward function. The second formula is: in, Indicates the target strategy With the jth training strategy The original reward function obtained by adversarial represents the target reward function, is the network model weight coefficient of the learner, Represents the diversity score.

6. The intelligent decision-making method based on a distributed training framework according to claim 1, characterized in that: The aggregating the network model parameters to obtain the global model parameters includes: The global model parameters are calculated according to the third formula, which is: in, represents the global model parameters, represents the network model parameters of the jth learner, k represents the number of learners, and also represents k groups of data.

7. The intelligent decision-making method based on a distributed training framework according to claim 1, characterized in that: Each training strategy corresponds to a set of parallel distributed simulation interaction environments, one or more samplers, and a learner.

8. An intelligent decision-making system based on a distributed training framework, characterized in that: The distributed training framework includes a sampler, a learner and a coordination server, and there are multiple samplers and learners; The intelligent decision-making system comprises: A strategy selection module, used to select k strategies from a first strategy set as training strategies, and determine a sampler and a learner corresponding to each training strategy; the first strategy set is a strategy set initialized in a game environment; A sampling module, used for performing adversarial sampling on each training strategy and a target strategy as a group to obtain first sampling data, wherein the target strategy is a strategy in the first strategy set, and the target strategy is different from the k training strategies; an aggregation module, configured to train a learner based on the first sampling data to obtain network model parameters of the learner, and aggregate the network model parameters to obtain global model parameters; The iterative training module is used to send the global model parameters to each learner and each sampler for iterative training to obtain a target strategy set that meets a preset convergence condition.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images