Method, device, equipment, storage medium and product for training scheduling agent

Abstract

This application relates to a training method, apparatus, device, storage medium, and product for a scheduling agent. The method includes: determining parent adversarial scenario pairs from a pre-set adversarial memory; performing evolutionary processing on each parent adversarial scenario in the parent adversarial scenario pairs to generate multiple candidate adversarial scenarios; obtaining the scheduling agent's cognitive entropy for each candidate adversarial scenario by interacting with the environment in each candidate adversarial scenario; the cognitive entropy characterizes the scheduling agent's cognitive uncertainty regarding the candidate adversarial scenarios; determining a target adversarial scenario from the candidate adversarial scenarios based on the cognitive entropy of each candidate adversarial scenario, and updating the scheduling agent's internal parameters based on the target adversarial scenario. This method can improve the generalization ability and robustness of the scheduling agent's scheduling strategy.

Description

Technical Field

[0001] This application relates to the field of scheduling technology, and in particular to a training method, apparatus, device, storage medium, and product for a scheduling agent. Background Technology

[0002] In complex systems such as modern logistics, unmanned system cluster collaboration, and cloud computing resource allocation, efficient and reliable scheduling is the key to ensuring the overall performance of the system.

[0003] In related technologies, deep reinforcement learning is often applied to the scheduling field, modeling the scheduling system as an interaction process between an agent and the environment. The scheduling agent can autonomously find a scheduling strategy that can maximize long-term cumulative rewards through continuous trial and error learning.

[0004] However, the agent-based scheduling methods in related technologies are highly dependent on the training scenarios in the agent training environment. The diversity of training scenarios is limited, which leads to insufficient generalization ability and robustness of agent scheduling strategies. Summary of the Invention

[0005] Therefore, it is necessary to provide a training method, apparatus, device, storage medium, and product for a scheduling agent to address the aforementioned technical problems and improve the generalization ability and robustness of the scheduling agent's scheduling strategy.

[0006] Firstly, this application provides a method for training a scheduling agent, the method comprising:

[0007] Determine the parent adversarial scenario pair from the pre-set adversarial memory bank;

[0008] The parent adversarial scenarios in the parent adversarial scenario pair are evolved to generate multiple candidate adversarial scenarios;

[0009] By scheduling agents to interact with the environment in various candidate adversarial scenarios, the cognitive entropy of the scheduling agents regarding each candidate adversarial scenario is obtained; cognitive entropy represents the cognitive uncertainty of the scheduling agents regarding the candidate adversarial scenarios.

[0010] Based on the cognitive entropy of each candidate adversarial scenario, the target adversarial scenario is determined from each candidate adversarial scenario, and the internal parameters of the scheduling agent are updated based on the target adversarial scenario.

[0011] In one embodiment, each adversarial scenario corresponds to a feature vector; the parent adversarial scenarios in the parent adversarial scenario pair are subjected to evolutionary processing to generate multiple candidate adversarial scenarios, including:

[0012] The elements in the feature vectors of each parent adversarial scenario that are in the same intersection interval are swapped to obtain the child adversarial scenario corresponding to each parent adversarial scenario; the intersection interval is determined based on two random intersection points.

[0013] Add perturbation values ​​to the feature vectors of each sub-generation adversarial scenario to generate candidate sub-generation adversarial scenarios for each sub-generation adversarial scenario;

[0014] The candidate adversarial scenarios for each generation of adversarial scenarios are summarized to obtain multiple candidate adversarial scenarios.

[0015] In one embodiment, the number of scheduling agents is one; by having the scheduling agent interact with the environment in each candidate adversarial scenario, the cognitive entropy of the scheduling agent regarding each candidate adversarial scenario is obtained, including:

[0016] For any candidate adversarial scenario, input the candidate adversarial scenario into the scheduling training environment, trigger the scheduling agent to interact with the candidate adversarial scenario in the environment, and obtain the information entropy of the scheduling agent at each time step;

[0017] By aggregating the information entropy of the scheduling agent at each time step, we obtain the cognitive entropy of the scheduling agent regarding the candidate adversarial scenario.

[0018] In one embodiment, obtaining the information entropy of the scheduling agent at each time step includes:

[0019] For any given time step, obtain the probability of the scheduling agent choosing multiple states at that time step, and perform a logarithmic operation on each state choice probability.

[0020] The logarithmic result of each state selection probability is multiplied with the corresponding state selection probability to obtain the quantified value of the uncertainty of multiple states of the scheduling agent at the time step.

[0021] The information entropy of the scheduling agent at each time step is obtained by superimposing the quantified values ​​of multiple state uncertainties at each time step.

[0022] In one embodiment, the target adversarial scenario is determined from the candidate adversarial scenarios based on the cognitive entropy of each candidate adversarial scenario, including:

[0023] The candidate adversarial scenarios are sorted in descending order of cognitive entropy to obtain a sequence of candidate adversarial scenarios.

[0024] Based on the average rate of change of external reward of the scheduling agent over a preset number of historical iteration cycles, target adversarial scenarios are selected from the candidate adversarial scenario sequence.

[0025] In one embodiment, a target adversarial scenario is selected from a sequence of candidate adversarial scenarios based on the average rate of change of extrinsic rewards of the scheduling agent over a preset number of historical iteration cycles, including:

[0026] If the average rate of change of external reward is less than a preset lower threshold, the first N candidate adversarial scenarios in the candidate adversarial scenario sequence are determined as the target adversarial scenario; N is an integer greater than 1.

[0027] When the average rate of change of external reward is greater than a preset upper limit threshold, the median of the cognitive entropy of each candidate adversarial scenario is calculated, and the candidate adversarial scenario corresponding to the cognitive entropy adjacent to the median is selected from the candidate adversarial scenario sequence as the target adversarial scenario.

[0028] In one embodiment, updating the internal parameters of the scheduling agent based on the target adversarial scenario includes:

[0029] The scheduling agent is trained using a target adversarial scenario to obtain the target training trajectory of the scheduling agent in the target adversarial scenario;

[0030] The target training trajectory and the historical training trajectory of the scheduling agent are used as the training trajectory set;

[0031] Based on the training trajectory set, the parameters of the scheduling agent are updated in batches until the convergence condition is met, thus obtaining a well-trained scheduling agent.

[0032] In one embodiment, the scheduling agent is trained using a target adversarial scenario to obtain the target training trajectory of the scheduling agent in the target adversarial scenario, including:

[0033] The scheduling agent acquires its scheduling decisions, cognitive entropy, and environmental reward signals at each time step in a target adversarial scenario; the environmental reward signals are determined based on the environmental information of the scheduling agent's execution of the scheduling strategy.

[0034] The cognitive entropy and environmental reward signal corresponding to each time step are weighted to obtain the combined reward signal for each time step.

[0035] By summarizing the scheduling decisions and combined reward information at each time step, the target training trajectory of the scheduling agent in the target adversarial scenario is obtained.

[0036] In one embodiment, the target adversarial scenario includes multiple scenarios; the method further includes:

[0037] Obtain the average cognitive entropy of each adversarial scenario in the adversarial memory bank;

[0038] The cognitive entropy of each adversarial scenario is compared with the mean cognitive entropy, and the adversarial memory is updated based on the comparison results.

[0039] In one embodiment, updating the adversarial memory based on the comparison results includes:

[0040] For any target adversarial scenario, if the cognitive entropy of the target adversarial scenario is greater than the average cognitive entropy, then the target adversarial scenario is added to the adversarial memory bank, and the adversarial scenario corresponding to the minimum cognitive entropy in the adversarial memory bank is deleted.

[0041] In one embodiment, the number of scheduling agents is multiple; the method further includes:

[0042] For any candidate adversarial scenario, the probability distribution of interaction information between each scheduling agent is obtained by the environmental interaction of each scheduling agent in the candidate adversarial scenario.

[0043] Based on the probability distribution of interactive information, the interaction information entropy between each scheduling agent is calculated; the interaction information entropy characterizes the cooperative state determinism of each scheduling agent in the candidate adversarial scenario.

[0044] The weighted sum of the interaction information entropy of each scheduling agent and the cognitive entropy of each scheduling agent regarding the candidate adversarial scenario yields the composite entropy of each scheduling agent regarding the candidate adversarial scenario.

[0045] Secondly, this application also provides a training apparatus for scheduling intelligent agents, comprising:

[0046] The adversarial scenario acquisition module is used to determine the parent adversarial scenario pair from the preset adversarial memory library;

[0047] The adversarial scenario evolution module is used to perform evolution processing on each parent adversarial scenario in the parent adversarial scenario pair to generate multiple candidate adversarial scenarios;

[0048] The cognitive entropy calculation module is used to obtain the cognitive entropy of the scheduling agent for each candidate adversarial scenario by scheduling the agent to interact with the environment in each candidate adversarial scenario; cognitive entropy represents the cognitive uncertainty of the scheduling agent for the candidate adversarial scenario.

[0049] The parameter update module is used to determine the target adversarial scenario from each candidate adversarial scenario based on the cognitive entropy of each candidate adversarial scenario, and update the internal parameters of the scheduling agent based on the target adversarial scenario.

[0050] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the methods in the first aspect.

[0051] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the methods in the first aspect.

[0052] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the methods in the first aspect.

[0053] The aforementioned training method, apparatus, device, storage medium, and product for the scheduling agent determine parent adversarial scenario pairs from a pre-set adversarial memory bank; perform evolutionary processing on each parent adversarial scenario in the parent adversarial scenario pair to generate multiple candidate adversarial scenarios; obtain the scheduling agent's cognitive entropy for each candidate adversarial scenario by interacting with the environment in each candidate adversarial scenario; cognitive entropy represents the scheduling agent's cognitive uncertainty regarding the candidate adversarial scenarios; based on the cognitive entropy of each candidate adversarial scenario, determine the target adversarial scenario from each candidate adversarial scenario, and update the scheduling agent's internal parameters based on the target adversarial scenario. In this method, by performing evolutionary processing on the parent adversarial scenarios in the parent adversarial scenario pair, multiple candidate adversarial scenarios are provided for the training of the scheduling agent to improve the diversity of training scenarios and fully explore adversarial scenarios that can deeply probe the cognitive and collaborative boundaries of the scheduling agent; then, based on the scheduling agent's cognitive uncertainty regarding each candidate adversarial scenario, a target adversarial scenario matching the scheduling agent is selected from each candidate adversarial scenario, and the scheduling agent's internal parameters are updated based on the target adversarial scenario, thereby achieving optimized updates to the scheduling agent. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 This is a schematic diagram of the modules of a scheduling agent training system in one embodiment;

[0056] Figure 2 This is a flowchart illustrating a training method for scheduling agents in one embodiment;

[0057] Figure 3 This is a flowchart illustrating the steps for determining candidate adversarial scenarios in one embodiment.

[0058] Figure 4 This is a flowchart illustrating the cognitive entropy determination step in one embodiment;

[0059] Figure 5 This is a flowchart illustrating the steps for determining the composite entropy in one embodiment;

[0060] Figure 6 This is a flowchart illustrating the steps involved in determining a target adversarial scenario in one embodiment.

[0061] Figure 7 This is a flowchart illustrating the parameter update steps of a scheduling agent in one embodiment;

[0062] Figure 8 This is a flowchart illustrating the target training trajectory determination steps in one embodiment;

[0063] Figure 9 This is a flowchart illustrating the memory update steps in one embodiment;

[0064] Figure 10 This is a flowchart illustrating the training method for scheduling agents in another embodiment;

[0065] Figure 11 This is a structural block diagram of a training device for scheduling agents in one embodiment;

[0066] Figure 12 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0068] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0069] In complex systems such as modern logistics, unmanned system cluster collaboration, and cloud computing resource allocation, efficient and reliable scheduling is crucial to ensuring overall system performance. Traditional scheduling methods often rely on mathematical programming or heuristic algorithms. While these methods perform well in handling static, deterministic problems, their rigidity and computational complexity often make them inadequate when facing high-dimensional, dynamically changing, and uncertain real-world environments.

[0070] In recent years, with the rapid development of deep reinforcement learning technology, its application to solving complex scheduling problems has become an important research direction. By modeling the scheduling system as an interaction process between an agent and its environment, the scheduling agent can autonomously find a scheduling strategy that maximizes long-term cumulative rewards through continuous trial and error learning. This end-to-end learning paradigm enables it to handle nonlinear, high-dimensional, and complex constraints that are difficult to model using traditional methods, demonstrating enormous application potential.

[0071] However, existing deep reinforcement learning-based scheduling methods still face significant challenges in practical applications. The training effectiveness of these methods is highly dependent on the diversity and challenge of the scenarios provided by their training environment. In typical practice, training scenarios are often generated through random parameters or rely on a fixed, pre-designed set of cases. The training data generated in this way has a relatively homogeneous distribution and is difficult to fully cover various critical edge cases or unexpected events that occur in the real world. Therefore, scheduling agents trained in this way, although performing well on the test set, often lack sufficient generalization ability and robustness when deployed in real-world environments. Once encountering novel or extreme scenarios not seen during training, they may make suboptimal or even catastrophic decisions.

[0072] Furthermore, existing training frameworks are not ideally efficient at improving the robustness of agents. Lacking effective mechanisms to proactively identify and exploit the agent's decision-making weaknesses, the training process is often indiscriminate, relying solely on massive, undifferentiated training to passively and accidentally cover scenarios that expose model flaws. This process is not only extremely time-consuming, but also lacks guaranteed improvement in learning effectiveness. When policy learning reaches a bottleneck, simply increasing the training volume often yields minimal results.

[0073] Furthermore, the reward mechanism design of current methods also has limitations. The agent's learning is entirely driven by an external reward signal directly related to the task objective, such as task completion time or resource utilization. This single source of reward ignores the agent's internal decision-making state. A scenario that can induce high uncertainty or cognitive ambiguity in the agent's decision-making has extremely high training value for improving its robustness, but this value cannot be reflected in traditional external reward signals. As a result, these valuable training opportunities are ignored, and the agent lacks the intrinsic motivation to explore and solve its own decision boundary problems.

[0074] In summary, the embodiments of this application provide a training method for scheduling agents, which can be applied to, for example... Figure 1 The scheduling agent training system shown. Figure 1This is a schematic diagram of the structure of a scheduling agent training system, including a scheduling agent 101, an adversarial memory bank 102, an adversarial evolution engine 103, an information entropy evaluation module 104, a scheduling training environment 105, and an adversarial scenario selection module 106. The adversarial evolution engine 103 generates candidate adversarial scenarios by performing evolutionary operations on the adversarial scenarios in the adversarial memory bank 102. The information entropy evaluation module 104 calculates the cognitive entropy of the scheduling agent 101, or the cognitive entropy of the scheduling agent 101 and the interaction information entropy between the agents, and merges them into a composite entropy score to quantify the degree of challenge that the scenario poses to the agent's decision-making and collaborative mechanisms.

[0075] The scheduling agent 101 is configured to generate a scheduling decision a based on an environmental observation o received from the scheduling training environment 105. The scheduling training environment 105 is connected to the scheduling agent 101 and is configured to execute the scheduling decision and return a new environmental observation and an external reward signal to the scheduling agent 101.

[0076] For example, in a cloud computing resource allocation scenario, environmental observation o may include information such as the number of tasks currently pending, the CPU utilization rate of each server, and the memory usage rate. The scheduling agent 101 generates a scheduling decision a based on these observations to allocate a specific task to a certain server. After the scheduling training environment 105 executes the decision, it will feed back the new server resource status as a new environmental observation and generate an external reward signal based on indicators such as task completion time and resource utilization rate. For example, the shorter the task completion time and the higher the resource utilization rate, the larger the value of the external reward signal.

[0077] In one embodiment, the internal decision-making model of the scheduling agent 101 employs a graph neural network. In a specific embodiment, a graph structure G=(V, E) is constructed based on the constituent elements of the scheduling problem. This construction process includes: defining each scheduling task or each schedulable resource in the scheduling problem as a node v in the graph structure G, with the set of all nodes v constituting the node set V; defining the constraint relationship between tasks and resources, or the connectivity relationship between resources, as an edge e connecting the corresponding nodes, with the set of all edges e constituting the edge set E; each node has an initial feature vector describing its inherent attributes and current state. Taking the logistics distribution scheduling problem as an example, scheduling tasks may include "transporting a batch of electronic products from warehouse A to store B" or "transporting fresh food from production site C to cold chain center D," etc. Each task is treated as a node v, and its initial feature vector... It can include information such as the weight and volume of the task cargo, transportation time limit, origin and destination; schedulable resources such as "delivery vehicle 1", "delivery vehicle 2" and "drone 3" are also treated as nodes v, and the feature vectors include information such as load capacity, range, current location and maintenance status; the constraints between tasks and resources, such as "the load capacity of delivery vehicle 1 can meet the transportation needs of electronic products" and "the range of drone 3 cannot cover the distance from the production site C to the cold chain center D", as well as the connectivity between resources, such as "delivery vehicle 1 and drone 3 can complete the cargo handover at the transfer station E", are all defined as edges e, thus constructing a complete graph structure G, which provides the foundation for feature extraction of graph neural networks.

[0078] In one specific embodiment, the graph neural network calculates the final embedding representation of each node by performing multi-layer message passing and feature aggregation operations. In each layer, a node aggregates the feature vectors of its first-order neighboring nodes and, combined with its own feature vector, updates its own embedding representation through a nonlinear transformation function. After multiple layers of operations, the calculated final embedding representation contains the high-order topological structure information of the node and its neighborhood. This final embedding representation is input into a policy network to calculate the probability distribution of the final scheduling decision. For example, in the graph structure of the logistics delivery scheduling described above, a "delivery vehicle" node, during the first layer of message passing, aggregates features such as cargo weight and transportation time limits of its neighboring "task" nodes and, combined with its own load capacity and range, updates its own embedding representation through a ReLU nonlinear transformation function. When entering the second layer, the node aggregates the updated features of its neighboring nodes from the first layer to further optimize its own embedding representation. After 3-5 layers of such operations, the final embedded representation of the "delivery vehicle" node not only includes its own attributes, but also incorporates higher-level information such as relevant task requirements and other collaborative resources. After inputting these embedded representations into the policy network, the network will output the probability distribution of the vehicle performing different tasks, such as "the probability of performing the electronic product transportation task is 60%", "the probability of performing the fresh food transportation task is 30%", and "the probability of being idle is 10%". The scheduling agent 101 selects the final scheduling decision based on this probability distribution.

[0079] The adversarial memory 102 is configured to store one or more adversarial scenarios. The adversarial evolution engine 103 is connected to the adversarial memory 102 and is configured to select parent adversarial scenarios from the adversarial memory 102 and perform evolutionary operations on the selected parent adversarial scenarios to generate a set of candidate adversarial scenarios. The adversarial evolution engine 103 outputs the candidate adversarial scenarios to the information entropy evaluation module 104. In practical applications, the adversarial memory 102 will create an index for each stored adversarial scenario, recording information such as the scenario generation time, the number of times it has been selected in history, and the corresponding composite entropy score, so that the adversarial evolution engine 103 can comprehensively consider the effectiveness and diversity of the scenario when selecting parent adversarial scenarios. For example, it will prioritize the selection of recently generated scenarios with high composite entropy scores and fewer selections as parents, avoiding the frequent use of old or ineffective scenarios, and ensuring the quality and diversity of the generated candidate adversarial scenarios.

[0080] The adversarial evolution engine 103 processes adversarial scenarios. An adversarial scenario is defined as a multi-dimensional feature vector, where each dimension corresponds to a perturbed environmental parameter, such as the communication interference intensity in a specific area, the failure probability of a specific resource node, or the attribute value of a sudden task. In one embodiment, the adversarial evolution engine 103 selects parent scenario pairs from the adversarial memory 102 and applies a single-point crossover operation to generate child vectors. Subsequently, one or more dimensions in the child vectors are perturbed with a preset mutation probability, ultimately generating a set of candidate adversarial scenarios. Taking the adversarial scenario of cloud computing resource allocation as an example, its feature vector dimensions may include "server 1 failure probability", "server 2 CPU load fluctuation range", "sudden task request frequency", "data transmission delay threshold", etc. Assuming that the parent scenario pairs selected from the adversarial memory 102 are A1=(0.2, The single-point crossover operation is performed after the third dimension (burst task request frequency) to generate offspring vectors A3=(0.2, 0.3, 5 times / minute, 10ms) and A4=(0.5, 0.1, 12 times / minute, 10ms). If the preset mutation probability is 0.2, random perturbation is performed on each dimension of the offspring vectors. For example, the "server 1 failure probability" dimension of A3 changes from 0.2 to 0.3 with a probability of 0.2, and the "burst task request frequency" dimension changes from 5 times / minute to 7 times / minute with a probability of 0.2. Finally, candidate adversarial scenarios A3=(0.3, 0.3, 7 times / minute, 8ms) and A4=(0.5, 0.1, 12 times / minute, 10ms) are generated (assuming that A4 does not mutate). This realizes the evolution and generation of adversarial scenarios.

[0081] The information entropy evaluation module 104 is connected to the scheduling agent 101 and the adversarial scenario selection module 106. For each candidate adversarial scenario received, the information entropy evaluation module 104 obtains its internal state from the scheduling agent 101 and calculates the cognitive entropy representing cognitive uncertainty based on the internal state.

[0082] Taking a single scheduling agent as an example, the action probability distribution output by the policy network of scheduling agent 101 under a given candidate scenario is obtained, and the entropy value is calculated based on the distribution. A higher cognitive entropy value corresponds to a probability distribution in which the probability values ​​of each action selection are more consistent.

[0083] Taking the collaborative scheduling of multiple scheduling agents as an example, the information entropy evaluation module 104 also calculates the inter-agent interaction information entropy, which characterizes the uncertainty of the collaborative state. Within a preset evaluation time window, the types of discrete interaction information units transmitted between multiple scheduling agents 101 are recorded, and the frequency of occurrence of each type of information unit is statistically analyzed. A higher inter-agent interaction information entropy value corresponds to a wider distribution of the types of interaction information units recorded within a time window. Further configuration involves weighted summation of cognitive entropy and inter-agent interaction information entropy to obtain a composite entropy score.

[0084] For example, taking the candidate adversarial scenario of unmanned system cluster collaboration, "communication interference intensity in a certain area is 0.6, and two unknown targets suddenly appear," when calculating cognitive entropy, the information entropy evaluation module 104 obtains the action probability distribution of the reconnaissance agent's policy network. If the distribution is "probability of continuing to reconnoiter the area is 35%", "probability of transferring to the surrounding area for reconnaissance is 30%", "probability of requesting support is 25%", and "probability of returning to the base is 10%", the probabilities are relatively close, and the cognitive entropy calculated according to the formula is high, indicating that the agent has a large uncertainty in its action selection in this scenario. If the probability distribution is "probability of continuing to reconnoiter the area is 80%" and "probabilities of other actions are all below 10%", then the cognitive entropy is low. When calculating the information entropy of interactions between intelligent agents, the preset evaluation time window is 15 seconds. During this period, the types of interactive information units transmitted between the reconnaissance agent, the attack agent, and the support agent are recorded, including "target location sharing", "attack request", "support route confirmation", "interference avoidance prompt", and "energy status feedback". The number of occurrences of each type of information unit are 5, 4, 3, 3 and 2 times respectively. The frequency of occurrence is relatively even and the distribution is wide. Therefore, the calculated information entropy of interactions between intelligent agents is high, reflecting the strong uncertainty of the cooperative state between intelligent agents in this scenario.

[0085] The information entropy evaluation module 104 outputs the calculated composite entropy score to the adversarial scenario selection module 106. In one embodiment, the cognitive entropy calculated by the information entropy evaluation module 104 is also output as an intrinsic reward signal to the scheduling agent 101. For example, when the agent's cognitive entropy is high in a candidate adversarial scenario, it indicates that the scenario poses a significant cognitive challenge to the agent. Using this cognitive entropy as an intrinsic reward signal can incentivize the agent to more proactively learn to cope with such high-challenge scenarios, optimize its decision-making model, and reduce cognitive uncertainty. If the cognitive entropy is low, it indicates that the agent's understanding of the scenario is relatively clear, and the intrinsic reward signal value is small, preventing the agent from over-focusing on simple scenarios and ignoring high-value challenging scenarios.

[0086] The adversarial scenario selection module 106 is connected to the information entropy evaluation module 104, the scheduling training environment 105, and the adversarial memory bank 102. The adversarial scenario selection module 106 is configured to select one or more adversarial scenarios from the candidate adversarial scenarios based on the received composite entropy score, and inject the selected adversarial scenarios into the scheduling training environment 105. At the same time, the adversarial scenario selection module 106 transmits the feature vectors of adversarial scenarios with high composite entropy scores to the adversarial memory bank 102 for dynamic updating of the adversarial memory bank 102. When selecting adversarial scenarios, the adversarial scenario selection module 106 first sorts the composite entropy scores of all candidate adversarial scenarios, and then determines the selection range in combination with the goal of the current training stage.

[0087] The adversarial scenario selection module 106 interacts with the adversarial memory bank 102. The non-monotonic selection strategy adopted by the adversarial scenario selection module 106 is implemented as follows in a specific embodiment: the system monitors the rate of change of the average external reward signal of the scheduling agent 101 in the past N training rounds. When the rate of change is lower than a preset lower threshold, the system enters the exploration phase. At this time, the adversarial scenario selection module 106 selects the scenario with the highest composite entropy score from the candidate scenarios for training. When the rate of change is higher than a preset upper threshold, the system enters the consolidation phase. At this time, the system selects the scenario with the composite entropy score in the preset low-to-medium range for training. In one specific embodiment, the dynamic update mechanism of the adversarial memory bank 102 is as follows: when the composite entropy score of an adversarial scenario is higher than the preset quantile (e.g., 75th percentile) of the scores of all scenarios in the memory bank 20, the scenario is added to the memory bank. If the memory bank capacity reaches the upper limit, the scenario with the lowest composite entropy score in the bank is removed. Assuming that the system presets N=10 training rounds, the lower limit threshold of the external reward signal change rate is 5%, the upper limit threshold is 20%, and the medium and low composite entropy score range is [0.3, 0.6]. If, over the past 10 training rounds, the agent's average external reward increases from 100 to 104 points (a change rate of 4%), which is below the lower threshold of 5%, the system enters the exploration phase. At this point, the scenario with the highest composite entropy score (e.g., 0.9) is selected from the candidate scenarios and injected into the training environment, forcing the agent to face highly challenging scenarios and overcome decision-making weaknesses. If, the average external reward increases from 100 to 122 points (a change rate of 22%), which is above the upper threshold of 20%, the system enters the consolidation phase. Scenarios with a composite entropy score between 0.3 and 0.6 (e.g., 0.45) are selected for training to help the agent consolidate its existing decision-making abilities and avoid degradation due to overexploration of highly challenging scenarios. For updating the adversarial memory bank 102, assuming the composite entropy scores of all scenarios in the current bank are 0.2, 0.4, 0.5, 0.7, 0.8, and the 75th percentile is 0.7, if the composite entropy score of a new scenario is 0.75, which is higher than 0.7, then it is added to the memory bank. At this time, the number of scenarios in the bank becomes 6. If the maximum capacity of the memory bank is 5, then the scenario corresponding to the lowest score of 0.2 is removed to ensure that the memory bank stores only high-quality adversarial scenarios.

[0088] For example, if the current training phase aims to "overcome the agent's decision-making weaknesses", 1-3 scenarios will be randomly selected from the top 20% of high composite entropy score scenarios and injected into the training environment; if the goal is to "consolidate the agent's existing decision-making capabilities", then scenarios with medium composite entropy score scenarios from the middle 40%-60% of the ranking will be selected.

[0089] In addition, to avoid repeatedly selecting the same or similar scenes, the information entropy evaluation module 104 will also compare the candidate scene with the scene feature vectors that have been recently injected into the training environment. If the similarity exceeds a preset threshold (such as 85%), the scene will be skipped to ensure the diversity of training scenes.

[0090] In one exemplary embodiment, such as Figure 2 As shown, a method for training a scheduling agent is provided, the method comprising:

[0091] S201, determine the parent adversarial scenario pair from the preset adversarial memory bank.

[0092] The adversarial memory includes multiple adversarial scenarios to provide the scheduling agent with a variety of adversarial situations. The scheduling agent generates scheduling strategies through reinforcement learning in these adversarial scenarios, thereby continuously optimizing its internal parameters to make better scheduling decisions.

[0093] Optionally, the adversarial memory includes the scene generation time, historical selection count, and composite entropy score for each adversarial scenario. The composite entropy score quantifies the scheduling agent's cognitive uncertainty regarding the adversarial scenario. A higher composite entropy score indicates less accurate decisions made by the scheduling agent in that scenario, thus increasing the training value of using that scenario to train the scheduling agent. In this case, two recently generated adversarial scenarios with high composite entropy scores and low selection counts can be selected as a parent adversarial scenario pair to avoid frequently using old or ineffective scenarios and ensure the quality and diversity of generated candidate adversarial scenarios.

[0094] Optionally, the adversarial memory includes the composite entropy score for each adversarial scenario. In this case, the composite entropy scores for each adversarial scenario can be summed to obtain the total composite entropy score; then, the ratio between the composite entropy score of each adversarial scenario and the total composite entropy score is used to determine the selection probability for each adversarial scenario, and each selection probability is mapped to the interval [0,1] to obtain the selection probability interval for each adversarial scenario; subsequently, a value between 0 and 1 is randomly generated as the first target selection probability, and the adversarial scenario corresponding to the selection probability interval to which the first target selection probability belongs is determined as a parent adversarial scenario; finally, a value between 0 and 1 is randomly generated again as the second target selection probability, and the adversarial scenario corresponding to the selection probability interval to which the second target selection probability belongs is determined as another parent adversarial scenario. The first target selection probability and the second target selection probability belong to different selection probability intervals.

[0095] For example, the adversarial memory bank includes adversarial scenarios A, B, and C. The composite entropy score of adversarial scenario A is 0.8, that of adversarial scenario B is 0.6, and that of adversarial scenario C is 0.6, with a total composite entropy score of 2.0. Therefore, the selection probability of A is 40%, while that of B and C is 30% each. The selected scenario is determined by randomly generating a value between 0 and 1. A is selected if the value is between 0 and 0.4, B is selected if the value is between 0.4 and 0.7, and C is selected if the value is between 0.7 and 1.0. This ensures that scenarios with higher scores have a higher probability of being selected as the parent adversarial scenario.

[0096] S202 involves performing evolutionary processing on each parent adversarial scenario in the parent adversarial scenario pair to generate multiple candidate adversarial scenarios.

[0097] Evolutionary processing refers to replacing, extending, and upgrading scene elements in each parent confrontation scene based on the core elements in each parent confrontation scene, thereby generating multiple candidate confrontation scenes that are different from the parent confrontation scenes.

[0098] Optionally, the element values ​​of the same scene element in each parent adversarial scene are superimposed or replaced to obtain two child adversarial scenes. Then, perturbation values ​​are randomly added to each child adversarial scene to obtain candidate adversarial scenes under each child adversarial scene. Finally, the candidate adversarial scenes of each child adversarial scene are summarized to obtain multiple candidate adversarial scenes.

[0099] S203, by scheduling agents to interact with the environment in each candidate adversarial scenario, obtain the cognitive entropy of the scheduling agents for each candidate adversarial scenario; cognitive entropy represents the cognitive uncertainty of the scheduling agents for the candidate adversarial scenario.

[0100] For any candidate adversarial scenario, the higher the cognitive entropy, the stronger the cognitive uncertainty of the scheduling agent regarding the candidate adversarial scenario, which can be used as a fuzzy scenario for training the scheduling agent; the lower the cognitive entropy, the clearer the decision of the scheduling agent regarding the candidate adversarial scenario, which can be used as a consolidation scenario for training the scheduling agent.

[0101] For any candidate adversarial scenario, the candidate adversarial scenario is input into the training environment of the scheduling agent, triggering the scheduling agent to interact with the environment in the candidate adversarial scenario. During the interaction, the internal decision model of the scheduling agent adopts a graph neural network to extract features from the topology of the scheduling training environment. In this embodiment, based on the internal decision model of the scheduling agent, the internal state data of the scheduling agent is collected, such as the hidden state of the decision model and the probability distribution of the policy network output. Then, based on the internal state data, the cognitive uncertainty of the scheduling agent in the candidate adversarial scenario is quantitatively evaluated to obtain the corresponding cognitive entropy.

[0102] As an extended embodiment, the agent interaction process in a single candidate adversarial scenario can be repeated (e.g., 10 times) to collect multiple sets of internal state data, calculate the average value of the probability distribution as the internal state data of the scheduling agent, and then substitute the average value of the probability distribution into the cognitive entropy calculation formula to obtain the cognitive entropy of the scheduling agent for the candidate adversarial scenario.

[0103] S204: Based on the cognitive entropy of each candidate adversarial scenario, determine the target adversarial scenario from each candidate adversarial scenario, and update the internal parameters of the scheduling agent based on the target adversarial scenario.

[0104] Optionally, candidate adversarial scenarios with cognitive entropy greater than a preset threshold can be identified as target adversarial scenarios. Alternatively, adversarial scenarios with high cognitive entropy and those with low cognitive entropy can be alternately selected at different training stages to train the scheduling agent and update its internal parameters.

[0105] In this embodiment, parent adversarial scenario pairs are determined from a preset adversarial memory bank; each parent adversarial scenario in the parent adversarial scenario pair undergoes evolutionary processing to generate multiple candidate adversarial scenarios; a scheduling agent interacts with the environment in each candidate adversarial scenario to obtain the scheduling agent's cognitive entropy for each candidate adversarial scenario; cognitive entropy represents the scheduling agent's cognitive uncertainty regarding the candidate adversarial scenarios; based on the cognitive entropy of each candidate adversarial scenario, a target adversarial scenario is determined from the candidate adversarial scenarios, and the internal parameters of the scheduling agent are updated based on the target adversarial scenario. In this method, by performing evolutionary processing on the parent adversarial scenarios in the parent adversarial scenario pairs, multiple candidate adversarial scenarios are provided for the training of the scheduling agent, thereby improving the diversity of training scenarios and fully exploring adversarial scenarios that can deeply probe the cognitive and collaborative boundaries of the scheduling agent; then, based on the scheduling agent's cognitive uncertainty regarding each candidate adversarial scenario, a target adversarial scenario matching the scheduling agent is selected from the candidate adversarial scenarios, and the internal parameters of the scheduling agent are updated based on the target adversarial scenario, thus achieving optimized updates to the scheduling agent.

[0106] In an exemplary embodiment, each adversarial scenario corresponds to a feature vector; an implementation of step S202, "evolutionary processing of each parent adversarial scenario in the parent adversarial scenario pair to generate multiple candidate adversarial scenarios," will be described, such as... Figure 3 As shown, it includes:

[0107] S301, swap the elements in the same intersection interval in the feature vectors of each parent adversarial scenario to obtain the child adversarial scenario corresponding to each parent adversarial scenario.

[0108] The intersection interval is determined based on two random intersection points.

[0109] Two intersection points are randomly selected, and the interval between the two intersection points is taken as the intersection interval (e.g., left open and right closed). Then, the feature vectors of the two parent adversarial scenarios in the same intersection interval are segmented and swapped to obtain the child adversarial scenarios corresponding to each parent adversarial scenario.

[0110] Optionally, the two parent adversarial scenarios in the parent adversarial scenario pair are A5=(a1,a2,a3,a4) and A6=(b1,b2,b3,b4). Randomly select the intersection points as 2 and 4, then the intersection interval is (2,4]. The two subsequent child adversarial scenarios are A7=(a1,a2,b3,b4) and A8=(b1,b2,a3,a4).

[0111] S302, add perturbation values ​​to the feature vectors of each sub-adversarial scenario to generate sub-candidate adversarial scenarios for each sub-adversarial scenario.

[0112] For any child adversarial scenario, perturbations are added to the elements of each feature vector of the child adversarial scenario according to a Gaussian distribution (e.g., mean 0, standard deviation 0.1).

[0113] Using a resource allocation and scheduling scenario as an example, the sub-adversarial scenario is represented by a four-dimensional vector, which respectively represents the probability of server 1 failure, the frequency of sudden task requests, and the failure time. The sub-adversarial scenario can be represented by (0.2, 0.3, 5 times / minute, 8ms). If the "probability of server 1 failure" dimension changes from 0.2 to 0.3 with a probability of 0.2, and the "frequency of sudden task requests" dimension changes from 5 times / minute to 7 times / minute with a probability of 0.2, then the sub-adversarial candidate scenarios of the sub-adversarial scenario can be represented by (0.3, 0.3, 7 times / minute, 8ms). Of course, different perturbation values ​​can be added to a single sub-adversarial scenario so that one sub-adversarial scenario corresponds to multiple sub-adversarial candidate scenarios, thereby improving the richness of the candidate adversarial scenarios.

[0114] S303 summarizes the candidate adversarial scenarios of each sub-generation adversarial scenario to obtain multiple candidate adversarial scenarios.

[0115] In this embodiment, the adversarial scenarios in the parent adversarial scenario pair are sequentially cross-segmented and randomly perturbed. The resulting multiple candidate adversarial scenarios retain the inheritance of the core adversarial logic of the parent scenario, and introduce the combination diversity of scenario features through cross-exchange, thereby achieving diversified expansion and differentiated coverage of candidate scenarios.

[0116] In practical scheduling scenarios, a single scheduling agent may be needed to generate a scheduling strategy, or multiple scheduling agents may need to work together to generate the optimal scheduling strategy. The following sections explain how to quantify the cognitive uncertainty of a scheduling agent regarding candidate adversarial scenarios in both single-agent and multi-agent training scenarios.

[0117] In one exemplary embodiment, the number of scheduling agents is one; such as Figure 4 As shown, by scheduling agents to interact with the environment in various candidate adversarial scenarios, the cognitive entropy of the scheduling agents regarding each candidate adversarial scenario is obtained, including:

[0118] S401, For any candidate adversarial scenario, input the candidate adversarial scenario into the scheduling training environment, trigger the scheduling agent to interact with the candidate adversarial scenario in the environment, and obtain the information entropy of the scheduling agent at each time step.

[0119] The candidate adversarial scenarios are input into the scheduling training environment, triggering the scheduling agent to interact with the scenario. This interaction is essentially a process of generating scheduling policies over multiple time steps. Based on this, the action distribution probability of the scheduling agent at each time step can be obtained. Then, the cognitive uncertainty of the scheduling agent at each time step can be quantitatively evaluated based on the action distribution probability, yielding the information entropy at each time step.

[0120] For example, obtaining the information entropy of the scheduling agent at each time step includes:

[0121] For any time step, obtain the multiple state selection probabilities of the scheduling agent at the time step, and perform a logarithmic operation on each state selection probability; multiply the logarithmic result of each state selection probability with the corresponding state selection probability to obtain the multiple state uncertainty quantification values ​​of the scheduling agent at the time step; and superimpose the multiple state uncertainty quantification values ​​at the time step to obtain the information entropy of the scheduling agent at the time step.

[0122] The scheduling agent has s at this time step i There are states, and the probability of choosing a state for each state is p(s). i The information entropy of the scheduling agent at each time step. The expression is as follows:

[0123]

[0124] In real-world scenarios, s i p(s) represents one dimension or a discrete state of the hidden state vector of the internal decision-making model of the scheduling agent. iLet be the probability of a dimension or state occurring at the current time step in the candidate adversarial scenario. Taking a logistics scheduling scenario as an example, if the hidden state vector of the internal decision model of the scheduling agent includes dimensions such as "task priority judgment", "transportation route selection", and "vehicle allocation tendency", when the candidate adversarial scenario is "a sudden traffic congestion in a certain area and multiple batches of high-priority goods arrive at the same time", the scheduling agent will collect the probability of judgment of each dimension state in this scenario. If the scheduling agent's selection probability of states such as "taking the highway detour", "taking the urban expressway", and "waiting for the congestion to ease" in the "transportation route selection" dimension is relatively close, all around 30%, then the cognitive entropy calculated according to the above formula will be high, indicating that the scheduling agent's cognitive uncertainty in transportation route selection in this scenario is large.

[0125] In this embodiment, the information entropy of each time step is obtained by calculating and superimposing the product of the logarithm of the state selection probability and the corresponding probability of the scheduling agent at each time step. This can reflect the degree of decision hesitation and exploration tendency of the agent at different time steps in real time, and provide objective data support for evaluating the stability, rationality and optimization space of the agent's scheduling decision.

[0126] S402, aggregate the information entropy of the scheduling agent at each time step to obtain the cognitive entropy of the scheduling agent on the candidate adversarial scenario.

[0127] Optionally, the average information entropy of the scheduling agent at each time step is calculated to obtain the cognitive entropy of the scheduling agent regarding the candidate adversarial scenario.

[0128] Optionally, the information entropy of the scheduling agent at each time step is sorted, and the median of the information entropy of the scheduling agent at each time step is determined as the cognitive entropy of the scheduling agent on the candidate adversarial scenario.

[0129] In this embodiment, candidate adversarial scenarios are input into the scheduling training environment, triggering the scheduling agent to interact with the candidate adversarial scenarios in the environment, obtaining the information entropy of the scheduling agent at each time step, and aggregating the information entropy of the scheduling agent at each time step to obtain the cognitive entropy of the scheduling agent on the candidate adversarial scenarios, providing a basis for subsequent evaluation of the effectiveness of the candidate adversarial scenarios.

[0130] The above embodiments describe the method for obtaining the quantification value of the cognitive uncertainty of the agent regarding the candidate adversarial scenario, assuming a single scheduling agent participates in the scheduling process. Next, the method for obtaining the quantification value of the cognitive uncertainty of the agent regarding the candidate adversarial scenario will be described for the case where multiple scheduling agents participate collaboratively in the scheduling process. In one exemplary embodiment, the number of scheduling agents is multiple; such as... Figure 5 As shown, the method further includes:

[0131] S501: For any candidate adversarial scenario, the probability distribution of interaction information between the scheduling agents is obtained by having each scheduling agent interact with the environment in the candidate adversarial scenario.

[0132] The probability distribution of interaction information between scheduling agents refers to the probability distribution of interaction information at each time step, including the probability of transmitting various types of interaction information at the current time step.

[0133] S502, based on the probability distribution of interaction information, calculates the interaction information entropy between each scheduling agent; the interaction information entropy characterizes the cooperative state determinism of each scheduling agent in the candidate adversarial scenario.

[0134] Based on the probability distribution of the interaction information at each time step, the interaction information entropy between each scheduling agent at that time step is calculated. Then, the interaction information entropy at each time step is aggregated, for example, by calculating the mean or taking the median, to obtain the interaction information entropy between each scheduling agent in the candidate adversarial scenario.

[0135] For any given time step, each scheduling agent has m at that time step. i There are several types of interactive information, and the selection probability for each type of interactive information is p(m). i The interactive information entropy at that time step The expression is as follows:

[0136]

[0137] In real-world scenarios, m j p(m) represents a specific unit of interactive information, such as a particular communication message or a collaborative action. j The information entropy represents the probability of an interactive information unit appearing in the current candidate adversarial scenario. Taking an unmanned system swarm collaboration scenario as an example, assuming there is an agent A responsible for reconnaissance, an agent B responsible for attack, and an agent C responsible for support, interactive information units may include "A sends target location information to B," "B requests C to provide energy supplies," and "C sends support routes to A," etc. When the candidate adversarial scenario is "some communication frequency bands are interfered with and unknown targets are discovered," the information entropy evaluation module 40 will count the number of various interactive information units transmitted between A, B, and C within a preset time window (e.g., 10 seconds). If, during this period, the number of times various interactive information units such as "A sends target location information to B," "A sends target location information to C," "B requests A to reconfirm the target," and "C asks B if it needs supplies" appears relatively evenly, with the probability of each information unit appearing between 20% and 25%, then the calculated inter-agent interactive information entropy will be high, indicating that in this adversarial scenario, the collaborative state between the agents is highly uncertain and lacks a stable and efficient interaction mode.

[0138] S503, the interaction information entropy of each scheduling agent and the cognitive entropy of each scheduling agent on the candidate adversarial scenario are weighted and summed to obtain the composite entropy of each scheduling agent on the candidate adversarial scenario.

[0139] The cognitive entropy of each scheduling agent regarding the candidate adversarial scenario can be found in the above embodiments for the case of a single scheduling agent participating in scheduling. The method for obtaining the quantification value of the agent's cognitive uncertainty regarding the candidate adversarial scenario will not be repeated here.

[0140] Furthermore, the cognitive entropy of each scheduling agent regarding the candidate adversarial scenario can be aggregated to obtain an aggregated cognitive entropy. Then, the interaction information entropy and the aggregated cognitive entropy are weighted and summed to obtain the composite entropy of each scheduling agent regarding the candidate adversarial scenario. The corresponding expression is as follows:

[0141]

[0142] in, and These are respectively aggregated cognitive entropy and interactive information entropy The preset weighting coefficients.

[0143] Furthermore, in multi-agent collaborative scheduling scenarios, the target adversarial scenario can be determined from each candidate adversarial scenario based on the composite entropy of each candidate adversarial scenario, and the internal parameters of each scheduling agent can be uniformly updated based on the target adversarial scenario.

[0144] In this embodiment, the probability distribution of interaction information of multiple scheduling agents in candidate adversarial scenarios is obtained and the interaction information entropy is calculated. Then, the composite entropy is obtained by weighted summation with the cognitive entropy of each agent on the scenario. From the two dimensions of individual cognitive uncertainty and group collaborative certainty, the cognition of the agent group on the candidate adversarial scenario is comprehensively and quantitatively evaluated, providing a basis for subsequent evaluation of the effectiveness of the candidate adversarial scenario.

[0145] Next, taking single-agent execution scheduling as an example, we will explain one possible way to implement the aforementioned step S204, "determining the target adversarial scenario from each candidate adversarial scenario based on the cognitive entropy of each candidate adversarial scenario," as follows: Figure 6 As shown, it includes:

[0146] S601, sort the candidate adversarial scenarios according to the order of cognitive entropy from largest to smallest to obtain the candidate adversarial scenario sequence.

[0147] Candidate adversarial scenarios are arranged in descending order of cognitive entropy, with high cognitive entropy scenarios first and low cognitive entropy scenarios last, to obtain a sequence of candidate adversarial scenarios.

[0148] S602, based on the average rate of change of external reward of the scheduling agent in a preset number of historical iteration cycles, select the target adversarial scenario from the candidate adversarial scenario sequence.

[0149] It should be noted that the scheduling agent is also continuously trained through multiple rounds of iterative training based on adversarial scenarios in the adversarial memory. Based on this, the extrinsic reward values ​​of the scheduling agent over a preset number of historical iteration cycles can be obtained, and the rate of change of multiple sets of adjacent extrinsic reward values ​​can be calculated to obtain the average extrinsic reward rate of change. Taking a 10-round iteration cycle as an example, the rate of change of change between every two adjacent extrinsic reward values ​​in the 10-round iteration cycle is calculated, corresponding to 9 rates of change. Then, the average of these 9 rates of change is calculated to obtain the average extrinsic reward rate of change.

[0150] Furthermore, the average rate of change of external reward is compared with the boundary value of the preset threshold range of the rate of change of external reward, and the target adversarial scenario is selected from the candidate adversarial scenario sequence based on the comparison result.

[0151] In an exemplary embodiment, if the average rate of change of external reward is less than a preset lower threshold, the first candidate adversarial scenario in the candidate adversarial scenario sequence is determined as the target adversarial scenario; if the average rate of change of external reward is greater than a preset upper threshold, the last candidate adversarial scenario in the candidate adversarial scenario sequence is determined as the target adversarial scenario.

[0152] In an exemplary embodiment, selecting a target adversarial scenario from a sequence of candidate adversarial scenarios based on the average rate of change of extrinsic rewards of the scheduling agent over a preset number of historical iteration cycles includes:

[0153] If the average rate of change of external reward is less than a preset lower threshold, the first N candidate adversarial scenarios in the candidate adversarial scenario sequence are determined as the target adversarial scenario; N is an integer greater than 1.

[0154] When the average rate of change of external reward is greater than a preset upper limit threshold, the median of the cognitive entropy of each candidate adversarial scenario is calculated, and the candidate adversarial scenario corresponding to the cognitive entropy adjacent to the median is selected from the candidate adversarial scenario sequence as the target adversarial scenario.

[0155] If the average rate of change of external reward is less than the preset lower limit threshold, it means that the scheduling agent has a relatively clear understanding of the scenarios in the current adversarial memory. In order to further optimize the scheduling agent, a candidate adversarial scenario with higher cognitive entropy can be selected as the target adversarial scenario. To enhance the internal scheduling exploration logic of the scheduling agent, the first N (e.g., the first 2) candidate adversarial scenarios in the candidate adversarial scenario sequence are determined as the target adversarial scenarios.

[0156] If the average rate of change of external reward exceeds a preset upper threshold, it means that the scheduling agent's understanding of the scenarios in the current adversarial memory is relatively vague. Therefore, to further solidify the scheduling agent's understanding, candidate adversarial scenarios with moderate scenario awareness can be selected as target adversarial scenarios. The median of the cognitive entropy of each candidate adversarial scenario is calculated, and the candidate adversarial scenarios with cognitive entropies adjacent to the median are selected as target adversarial scenarios. For example, if the median cognitive entropy of all candidate adversarial scenarios is 0.65, three candidate adversarial scenarios with cognitive entropies between 0.52 and 0.78 (0.78, 0.65, 0.52) are selected as target adversarial scenarios.

[0157] In this embodiment, candidate adversarial scenarios are first sorted from highest to lowest cognitive entropy to obtain a sequence, clarifying the challenge level of each scenario to the agent. Then, combined with the average external reward change rate of the agent's historical iteration cycles, the target adversarial scenario corresponding to the agent's current training progress is accurately matched, ensuring that the training process focuses on high-value scenarios. Furthermore, target adversarial scenarios are dynamically selected based on the average external reward change rate of the agent's historical iteration cycles. When the reward change rate is below the lower threshold, the top N high-difficulty scenarios are selected to strengthen training and break through bottlenecks. When it is above the upper threshold, medium-difficulty scenarios with cognitive entropy close to the median are selected to consolidate the training effect. This achieves adaptive matching and precise selection of training scenarios, avoiding training stagnation due to overly difficult scenarios or inefficiency due to overly easy scenarios. It also dynamically adjusts the scenario difficulty according to the agent's current training progress, helping the agent steadily improve its decision-making ability and environmental adaptability, ensuring the efficiency and stability of the training process.

[0158] In one exemplary embodiment, such as Figure 7 As shown, the internal parameters of the scheduling agent are updated based on the target adversarial scenario, including:

[0159] S701 trains the scheduling agent using a target adversarial scenario to obtain the target training trajectory of the scheduling agent in the target adversarial scenario.

[0160] For any adversarial scenario, the adversarial scenario is input into the scheduling training environment, triggering the scheduling agent to interact with the adversarial scenario. The state selection probability distribution, scheduling action, and environmental reward signal fed back by the scheduling training environment are obtained at each time step in the adversarial scenario. Then, the cognitive entropy of the scheduling agent at each time step is determined based on the state selection probability distribution. Based on the cognitive entropy, scheduling action, and environmental reward signal fed back by the scheduling training environment at each time step, the target training trajectory of the scheduling agent in the adversarial scenario is determined.

[0161] S702 uses the target training trajectory and the historical training trajectory of the scheduling agent as the training trajectory set.

[0162] The target training trajectory of the scheduling agent in the adversarial scenario, along with a preset number of historical training trajectories stored during the historical training phase, are aggregated to form the training trajectory set of the scheduling agent. For example, the target training trajectory and the most recent 999 historical training trajectories can be used as the training trajectory set.

[0163] The target training trajectory and the historical training trajectory of the scheduling agent can be stored in an experience replay buffer. The amount of data stored in the experience replay buffer is limited. When the amount of data exceeds the limit, the oldest data is removed in a first-in-first-out manner to ensure the timeliness of the training trajectory data in the experience replay buffer.

[0164] S703 updates the parameters of the scheduling agent in batches based on the training trajectory set until the convergence condition is met, thus obtaining a trained scheduling agent.

[0165] Based on the training trajectory set, 32-128 trajectory data points are randomly sampled as a batch. The ratio of the current policy to the old policy of the scheduling agent is calculated based on these training trajectories. A pruning operation is used, for example, limiting the ratio to [0.8, 1.2] to prevent excessive policy updates. Then, the policy loss is calculated, for example, the product of the pruned policy ratio and the advantage function. Simultaneously, the value of each state is predicted through a value network, and the mean squared error is calculated as the value loss along with the actual cumulative reward. The policy loss and value loss are weighted and summed, for example, with a policy loss weight of 0.7 and a value loss weight of 0.3, to obtain the total loss. A stochastic gradient descent (SGD) or adaptive moment estimation (Adam) optimizer is used to calculate the gradient based on the total loss. The parameters of the graph neural network, policy network, and other internal decision networks are updated with a preset learning rate (0.001), causing the scheduling agent's decision policy to iterate towards a better direction until the convergence condition is met, resulting in a trained scheduling agent.

[0166] The convergence conditions can be: the number of training iterations reaches a preset upper limit (e.g., 1000 times); the average combined reward signal of the scheduling agent reaches a preset target value (e.g., above 90 points) within 20 consecutive iteration cycles; and the decision success rate of the agent in multiple typical test scenarios (e.g., high load, sudden failure, complex collaborative scenarios) is stable at above 95%.

[0167] Compared to existing agent reinforcement learning methods based on adversarial training, the generation of adversarial scenarios usually relies on maximizing the loss function of the external task, which can easily lead to insufficient diversity of generated scenarios, pattern collapse, and an inability to effectively detect the agent's decision-making boundaries and the vulnerability of its collaborative mechanisms at the cognitive level. The embodiments of this application determine the total scheduling loss from two dimensions: policy loss and value loss, so as to effectively detect the agent's decision-making boundaries and collaborative mechanisms at the cognitive level.

[0168] In this embodiment, a target adversarial scenario is used to train the scheduling agent, and the target training trajectory of the scheduling agent in the target adversarial scenario is obtained. The target training trajectory and the historical training trajectory of the scheduling agent are used as a training trajectory set. Based on the training trajectory set, the parameters of the scheduling agent are updated in batches until the convergence condition is met, so that the trained scheduling agent has more reliable scheduling decision-making ability, stronger environmental adaptability and better performance in handling complex scenarios.

[0169] In an exemplary embodiment, one possible implementation of the aforementioned step S701, "training the scheduling agent using a target adversarial scenario and obtaining the target training trajectory of the scheduling agent in the target adversarial scenario," will be described, such as... Figure 8 As shown, it includes:

[0170] S801 acquires the scheduling agent's scheduling decisions, cognitive entropy, and environmental reward signals at each time step in a target adversarial scenario.

[0171] The environmental reward signal is determined based on the environmental information of the scheduling agent executing the scheduling strategy.

[0172] Among them, cognitive entropy is determined by the internal decision-making hidden state of the scheduling agent executing the scheduling strategy, and belongs to the intrinsic signal, while environmental reward signal is determined by the environmental information of the scheduling agent executing the scheduling strategy, and belongs to the environmental feedback external signal.

[0173] S802 weights the cognitive entropy and environmental reward signal corresponding to each time step to obtain the combined reward signal for each time step.

[0174] For time step t, the cognitive entropy corresponding to that time step and environmental reward signals We perform a weighted summation to obtain the combined reward signal. The corresponding expression is as follows:

[0175]

[0176] in, The weight value for the environmental reward signal.

[0177] Taking cloud computing resource allocation training as an example, after injecting the selected adversarial scenario of "server 1 failure probability 0.4, sudden task request frequency 8 times / minute" into the scheduling training environment, at time step t=1, the environment observes "5 new task requests, server 1 response latency increased". The agent generates the decision "assign 3 tasks to server 2 and 2 tasks to server 3". After the environment executes this decision, at time step t+1=2, it feeds back an external reward signal. (Due to timely task processing, resource utilization reached 85%); simultaneously, based on the agent's internal state at t=1, cognitive entropy was calculated. ,like Then combine reward signals .

[0178] S803 summarizes the scheduling decisions and combined reward information at each time step to obtain the target training trajectory of the scheduling agent in the target adversarial scenario.

[0179] The data from the agent's environmental observations, self-decision-making, and combined reward signals at each time step are used to form a complete training trajectory, providing a basis for subsequent parameter updates.

[0180] In this embodiment, the scheduling agent's scheduling decision, cognitive entropy, and environmental reward signal at each time step in the target adversarial scenario are obtained. Then, the cognitive entropy and environmental reward signal corresponding to each time step are weighted to obtain the combined reward signal of each time step. Finally, the scheduling decision and combined reward information of each time step are summarized to obtain the target training trajectory of the scheduling agent in the target adversarial scenario. This allows the target training trajectory to simultaneously carry the characteristics of decision-making behavior, environmental feedback, and decision uncertainty, providing a more accurate training basis for subsequent agent parameter updates.

[0181] In one exemplary embodiment, the target adversarial scenario includes multiple scenarios; such as Figure 9 As shown, the method also includes:

[0182] S901, obtain the average cognitive entropy of each adversarial scenario in the adversarial memory bank.

[0183] The adversarial memory bank includes multiple adversarial scenarios, each corresponding to a cognitive entropy. The average cognitive entropy of each adversarial scenario is calculated.

[0184] S902 compares the cognitive entropy of each target adversarial scenario with the mean cognitive entropy, and updates the adversarial memory bank based on the comparison results.

[0185] In one exemplary embodiment, target adversarial scenarios with cognitive entropy higher than the average cognitive entropy are added to the adversarial memory bank, or target adversarial scenarios with cognitive entropy lower than the average cognitive entropy are added to the adversarial memory bank.

[0186] In an exemplary embodiment, updating the adversarial memory bank based on the comparison results includes: for any target adversarial scenario, if the cognitive entropy of the target adversarial scenario is greater than the average cognitive entropy, then the target adversarial scenario is added to the adversarial memory bank, and the adversarial scenario corresponding to the minimum cognitive entropy in the adversarial memory bank is deleted.

[0187] If the cognitive entropy of the target adversarial scenario is greater than the average cognitive entropy, it means that the scheduling agent's understanding of the target adversarial scenario is relatively vague. In this case, the value of using the target adversarial scenario to train the scheduling agent is higher, so the target adversarial scenario is added to the adversarial memory. At this time, in order to avoid too many scenarios in the adversarial memory, the adversarial scenario corresponding to the minimum cognitive entropy in the adversarial memory is deleted simultaneously.

[0188] In this embodiment, by comparing the cognitive entropy of each target adversarial scenario with the average cognitive entropy of scenarios in the adversarial memory bank, the memory scenario bank is updated based on the comparison results. This not only continuously includes high-value, high-challenge scenarios with entropy higher than the average to enrich the difficult example samples in the bank, but also improves the effectiveness of the adversarial memory bank by filtering out or replacing inefficient scenarios with entropy lower than the average. Furthermore, target adversarial scenarios with cognitive entropy greater than the average cognitive entropy of the adversarial memory bank are included in the bank, while inefficient scenarios with the lowest cognitive entropy in the bank are simultaneously deleted. This maintains the high training gain attribute of scenarios in the memory scenario bank while avoiding resource consumption by inefficient scenarios.

[0189] In an exemplary embodiment, a collaborative training method for scheduling agents based on game-theoretic adversarial approaches is provided. The method begins with an initialization phase, where the parameters of the internal decision-making networks of one or more scheduling agents are randomly initialized. An initial set of seed adversarial scenarios is placed into the adversarial memory, and the system hyperparameters are preset. These hyperparameters include the learning rate of the reinforcement learning framework, the reward discount factor γ, and the intrinsic reward weight coefficient β used to calculate the combined reward signal. During initialization, the weights, biases, and other parameters of the internal decision-making networks (such as neural networks and policy networks) of the scheduling agents are processed according to a normal or uniform distribution. Random values ​​are assigned to ensure the network is unbiased in the initial state. The selection of initial seed adversarial scenarios is based on common challenges in real-world applications. For example, in logistics scheduling, seed scenarios may include transportation delays caused by severe weather, vehicle breakdowns, or the addition of emergency delivery tasks. The number of scenarios is typically set to 10-20 to provide a foundation for subsequent scenario evolution. Hyperparameter presets require multiple pre-experiments for adjustment. For instance, the learning rate of the reinforcement learning framework is initially set to 0.001, the reward discount factor γ is set to 0.9 (indicating a greater focus on recent rewards), and the intrinsic reward weight coefficient β is set to 0.5 to balance the influence of intrinsic and extrinsic rewards. After initialization, the system enters an iterative training process, such as... Figure 10 As shown, a complete training iteration includes the following steps:

[0190] S1001, Generate candidate adversarial scenarios.

[0191] The adversarial evolution engine selects parent scenario pairs from the adversarial memory based on the composite entropy score of the adversarial scenarios using a roulette wheel selection method. Subsequently, the adversarial evolution engine applies uniform crossover and Gaussian mutation operations to the feature vectors of the selected parent scenarios to generate a new set of candidate adversarial scenarios.

[0192] For example, first calculate the sum of the composite entropy scores of all scenarios, and then assign a selection probability proportional to the score to each scenario. If scenario A scores 0.8, scenario B scores 0.6, scenario C scores 0.6, and the total score is 2.0, then the selection probability of A is 40%, and that of B and C is 30% each. By randomly generating a value between 0 and 1, the selected scenario is determined. A is selected if the value is between 0 and 0.4, B is selected if the value is between 0.4 and 0.7, and C is selected if the value is between 0.7 and 1.0. This ensures that scenarios with higher scores have a higher probability of being selected as the parent.

[0193] S1002, Perform entropy value evaluation.

[0194] For each candidate adversarial scenario generated in step S1001, an evaluation is performed to calculate its corresponding cognitive entropy and inter-agent interaction information entropy, and finally, the composite entropy score of the candidate adversarial scenario is obtained.

[0195] During the evaluation process, candidate adversarial scenarios are first input into the scheduling training environment to trigger interactions between agents and the environment. Simultaneously, internal state data of the agents (such as hidden states of the decision model and probability distributions of the policy network output) and interaction data between multiple agents (such as communication message types and interaction frequencies) are collected in real time. For the calculation of cognitive entropy, the agent interaction process in the scenario is repeated multiple times (e.g., 10 times), collecting multiple sets of internal state data. The average value of the probability distribution is calculated and then substituted into the formula to reduce random errors. For the information entropy of interactions between agents, 3-5 different evaluation time windows (e.g., 5 seconds, 10 seconds, 15 seconds) are set to statistically analyze the frequency of interaction information units, and the average value is used as the final calculation basis. Finally, based on preset weight coefficients, the composite entropy score of each candidate scenario is calculated, and the score is associated with and stored with the scenario feature vector, providing data support for scenario selection in step S1003.

[0196] S1003. Select the adversarial scenario for training.

[0197] Obtain all candidate adversarial scenarios and their corresponding composite entropy scores. When the rate of change of the average external reward of the scheduling agent over the past multiple iterations is lower than a preset lower threshold, select one or more scenarios with the highest composite entropy scores; when the rate of change is higher than a preset upper threshold, select one or more scenarios with composite entropy scores near the median.

[0198] For example, with a preset 15 iteration cycles, a lower threshold of 8% and an upper threshold of 25% for the average rate of change in extrinsic reward, and the current candidate scene composite entropy scores ranked as 0.92, 0.85, 0.78, 0.65, 0.52, 0.43, and 0.38, if the agent's average extrinsic reward increases from 150 to 159 points in the past 15 iteration cycles (a change rate of 6%, which is lower than 8%), then the two scenes with the highest scores (0.92 and 0.85) are selected for training. If the average extrinsic reward increases from 150 to 190 points (a change rate of 26.7%, which is higher than 25%), then the median score is calculated to be 0.65, and the three scenes with scores between 0.52 and 0.78 (0.78, 0.65, and 0.52) are selected for training.

[0199] In addition, to prevent the selected scenes from being too similar, the module calculates the cosine similarity between the feature vectors of the selected scenes. If the similarity exceeds 90%, it is replaced with the next scene with a similar score, ensuring the diversity and effectiveness of the training scenes.

[0200] S1004, Perform agent training and reward calculation.

[0201] The adversarial scenario selected in step S1003 is injected into the scheduling training environment. At each time step, the environment returns an external reward signal and calculates an internal reward signal, and finally calculates a combined reward signal.

[0202] Taking cloud computing resource allocation training as an example, after injecting the selected adversarial scenario of "server 1 failure probability 0.4, sudden task request frequency 8 times / minute" into the scheduling training environment, at time step t=1, the environment observes "5 new task requests, server 1 response latency increased". The agent generates the decision "assign 3 tasks to server 2 and 2 tasks to server 3". After the environment executes this decision, at time step t+1=2, it feeds back an external reward signal. (Due to timely task processing, resource utilization reached 85%); simultaneously, based on the agent's internal state at t=1, cognitive entropy was calculated. ,like Then combine reward signals The scheduling agent records data such as environmental observations, its own decisions, and combined reward signals at each time step, forming a complete training trajectory and providing a basis for subsequent parameter updates.

[0203] S1005, Update parameters.

[0204] Trajectory data generated by the scheduling agent during interactions is stored in an experience replay buffer. A near-end policy optimization algorithm is used to sample data from the experience replay buffer, calculate policy loss and value loss, and then calculate the gradient based on this loss to update the internal decision network parameters of the scheduling agent. The experience replay buffer limits the amount of data it can store (e.g., a maximum of 10,000 trajectory data points). When the data volume exceeds this limit, the oldest data is removed according to the "first-in, first-out" principle to ensure the timeliness of the data in the buffer.

[0205] When employing the near-end policy optimization algorithm, 32-128 trajectory data points are randomly sampled from the experience replay buffer as a batch. Based on these data, the ratio of the current policy to the old policy is calculated. A pruning operation (e.g., limiting the ratio to [0.8, 1.2]) is used to prevent excessive policy updates. Then, the policy loss is calculated (e.g., the product of the pruned policy ratio and the advantage function). Simultaneously, the value of each state is predicted through the value network, and the mean squared error is calculated as the value loss along with the actual cumulative reward. The policy loss and value loss are weighted and summed (e.g., policy loss weight 0.7, value loss weight 0.3) to obtain the total loss. A stochastic gradient descent (SGD) or adaptive moment estimation (Adam) optimizer is used to calculate the gradient based on the total loss. The parameters of the internal decision networks, such as the graph neural network and policy network, are updated with a preset learning rate (e.g., 0.001) to guide the agent's decision-making policy towards a better direction.

[0206] S1006, Execute memory update.

[0207] The adversarial scenarios actually used in step S1004, along with their composite entropy scores calculated in step S1002, are compared with the scenarios already existing in the adversarial memory. If the composite entropy score of a scenario is higher than the average or median of all scenario scores in the memory, it is added to the memory, and the scenario with the lowest score is removed. For example, the adversarial scenario used in step S1004 has a composite entropy score of 0.8, and the current scenarios stored in the adversarial memory have scores of 0.3, 0.5, 0.6, 0.7, and 0.9, with an average of 0.6 and a median of 0.6. Since 0.8 is higher than the average and median, this scenario is added to the memory, which now contains 6 scenarios. If the memory capacity is limited to 5, the scenario corresponding to the lowest score of 0.3 is removed, and the updated memory scenario scores are 0.5, 0.6, 0.7, 0.8, and 0.9. At the same time, the memory bank will update the "last time" and "number of times" of each scene, providing more comprehensive reference information for the selection of parent scenes in the future, and ensuring that the memory bank always stores high-quality and high-value adversarial scenes.

[0208] Steps S1001 to S1006 constitute a complete closed-loop training iteration. This iterative process is repeated until one or more preset termination conditions are met. These preset termination conditions may include: the number of training iterations reaching a preset upper limit (e.g., 1000 times); the average combined reward signal of the scheduling agent reaching a preset target value (e.g., above 90 points) over 20 consecutive iteration cycles; and the agent's decision success rate remaining stable above 95% in multiple typical test scenarios (e.g., high load, sudden failure, complex collaborative scenarios). When any termination condition is met, the training process stops. At this point, the scheduling agent possesses strong robustness and adaptability and can be deployed to perform scheduling tasks in real-world application scenarios.

[0209] The game-theoretic adversarial scheduling agent collaborative training method provided in this application has the following beneficial effects: (1) By introducing an information entropy evaluation module, it is possible to quantify the cognitive uncertainty (cognitive entropy) of the scheduling agent in the candidate adversarial scenario and the uncertainty of the collaborative state of the multi-agent system (inter-agent interaction information entropy), and integrate them into a composite entropy score. This technical feature enables the system to generate and screen adversarial scenarios that are highly challenging and can deeply explore the cognitive and collaborative boundaries of agents, thereby avoiding the problems of insufficient scenario diversity and mode collapse caused by relying solely on external task losses in traditional methods.

[0210] (2) The calculated composite entropy score is used as the core basis for adversarial scenario selection and dynamic updating of adversarial memory. By selecting high-scoring adversarial scenarios for training and adding them to the adversarial memory, this invention establishes an adaptive evolution loop based on information entropy. This enables the system to continuously generate higher quality and more challenging adversarial scenarios, ensuring the robustness and effectiveness of the training process and improving the generalization ability of the finally trained agent.

[0211] (3) Cognitive entropy is used as the intrinsic reward signal in the policy learning process of the scheduling agent. This intrinsic reward signal is fused with the extrinsic reward signal from the environment to jointly guide the scheduling agent to optimize parameters. This mechanism enables the agent to actively explore and learn how to deal with uncertain scenarios at its own decision boundary, thereby fundamentally improving training efficiency and making the finally learned scheduling strategy exhibit higher robustness and adaptability when dealing with unknown and unexpected situations.

[0212] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0213] Based on the same inventive concept, this application also provides a training apparatus for a scheduling agent to implement the training method for the scheduling agent described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more training apparatus embodiments for scheduling agents provided below can be found in the limitations of the training method for scheduling agents described above, and will not be repeated here.

[0214] In one exemplary embodiment, such as Figure 11 As shown, a training device for scheduling intelligent agents is provided, including: an adversarial scenario acquisition module 1101, an adversarial scenario evolution module 1102, a cognitive entropy calculation module 1103, and a parameter update module 1104, wherein:

[0215] The adversarial scenario acquisition module 1101 is used to determine the parent adversarial scenario pair from the preset adversarial memory bank;

[0216] The adversarial scenario evolution module 1102 is used to perform evolution processing on each parent adversarial scenario in the parent adversarial scenario pair to generate multiple candidate adversarial scenarios.

[0217] The cognitive entropy calculation module 1103 is used to obtain the cognitive entropy of the scheduling agent for each candidate adversarial scenario by having the scheduling agent interact with the environment in each candidate adversarial scenario; the cognitive entropy represents the cognitive uncertainty of the scheduling agent for the candidate adversarial scenario.

[0218] The parameter update module 1104 is used to determine the target adversarial scenario from each candidate adversarial scenario based on the cognitive entropy of each candidate adversarial scenario, and update the internal parameters of the scheduling agent based on the target adversarial scenario.

[0219] In an exemplary embodiment, each adversarial scenario corresponds to a feature vector; the adversarial scenario evolution module 1102 includes: a cross-processing unit, a perturbation addition unit, and a scenario summarization unit, wherein:

[0220] The cross-processing unit is used to swap the elements in the feature vectors of each parent adversarial scenario that are in the same cross interval to obtain the child adversarial scenario corresponding to each parent adversarial scenario; the cross interval is determined based on two random cross points.

[0221] The perturbation addition unit is used to add perturbation values ​​to the feature vectors of each sub-adversarial scenario to generate sub-candidate adversarial scenarios for each sub-adversarial scenario.

[0222] The scenario aggregation unit is used to aggregate the candidate adversarial scenarios of each sub-generation adversarial scenario to obtain multiple candidate adversarial scenarios.

[0223] In an exemplary embodiment, the number of scheduling agents is one; the cognitive entropy calculation module 1103 includes: an information entropy determination unit and an information entropy aggregation unit, wherein:

[0224] The information entropy determination unit is used to input the candidate adversarial scenario into the scheduling training environment for any candidate adversarial scenario, trigger the scheduling agent to interact with the candidate adversarial scenario in the environment, and obtain the information entropy of the scheduling agent at each time step.

[0225] The information entropy aggregation unit is used to aggregate the information entropy of the scheduling agent at each time step to obtain the cognitive entropy of the scheduling agent on the candidate adversarial scenario.

[0226] In an exemplary embodiment, the information entropy determination unit is further configured to, for any time step, obtain multiple state selection probabilities of the scheduling agent at the time step, and perform logarithmic operation on each state selection probability; and multiply the logarithmic operation result of each state selection probability with the corresponding state selection probability to obtain the multiple state uncertainty quantification value of the scheduling agent at the time step.

[0227] The information entropy of the scheduling agent at each time step is obtained by superimposing the quantified values ​​of multiple state uncertainties at each time step.

[0228] In an exemplary embodiment, the parameter update module 1104 includes: a scene sorting unit and a scene determination unit, wherein:

[0229] The scenario sorting unit is used to sort the candidate adversarial scenarios in descending order of cognitive entropy to obtain a sequence of candidate adversarial scenarios.

[0230] The scenario determination unit is used to select target adversarial scenarios from the candidate adversarial scenario sequence based on the average rate of change of external rewards of the scheduling agent over a preset number of historical iteration cycles.

[0231] In an exemplary embodiment, the scenario determination unit is further configured to determine the top N candidate adversarial scenarios in the candidate adversarial scenario sequence as target adversarial scenarios when the average rate of change of external reward is less than a preset lower threshold; N is an integer greater than 1; when the average rate of change of external reward is greater than a preset upper threshold, calculate the median of the cognitive entropy of each candidate adversarial scenario, and select the candidate adversarial scenario corresponding to the cognitive entropy adjacent to the median from the candidate adversarial scenario sequence as the target adversarial scenario.

[0232] In an exemplary embodiment, the parameter update module 1104 further includes: a trajectory generation unit, a trajectory set generation unit, and a batch training unit, wherein:

[0233] The trajectory generation unit is used to train the scheduling agent in a target adversarial scenario and obtain the target training trajectory of the scheduling agent in the target adversarial scenario.

[0234] The trajectory set generation unit is used to take the target training trajectory and the historical training trajectory of the scheduling agent as the training trajectory set;

[0235] The batch training unit is used to update the parameters of the scheduling agent in batches based on the training trajectory set until the convergence condition is met, thus obtaining a trained scheduling agent.

[0236] In an exemplary embodiment, the trajectory generation unit is further configured to acquire the scheduling decision, cognitive entropy, and environmental reward signal of the scheduling agent at each time step in the target adversarial scenario; the environmental reward signal is determined based on the environmental information of the scheduling agent executing the scheduling strategy; the cognitive entropy and environmental reward signal corresponding to each time step are weighted to obtain the combined reward signal of each time step; the scheduling decision and combined reward information of each time step are summarized to obtain the target training trajectory of the scheduling agent in the target adversarial scenario.

[0237] In one exemplary embodiment, the target adversarial scenario includes multiple scenarios; the training device for scheduling intelligent agents further includes: a cognitive entropy mean calculation module and a memory bank update module, wherein:

[0238] The cognitive entropy mean calculation module is used to obtain the cognitive entropy mean of each adversarial scenario in the adversarial memory bank;

[0239] The memory bank update module is used to compare the cognitive entropy of each target adversarial scenario with the mean cognitive entropy, and update the adversarial memory bank based on the comparison results.

[0240] In an exemplary embodiment, the memory bank update module is further configured to, for any target adversarial scenario, add the target adversarial scenario to the adversarial memory bank if the cognitive entropy of the target adversarial scenario is greater than the average cognitive entropy, and delete the adversarial scenario corresponding to the minimum cognitive entropy in the adversarial memory bank.

[0241] In an exemplary embodiment, the number of scheduling agents is multiple; the training device for the scheduling agents further includes: a composite entropy determination module, used to obtain the probability distribution of interaction information between the scheduling agents by having each scheduling agent interact with the environment in the candidate adversarial scenario for any candidate adversarial scenario; calculate the interaction information entropy between the scheduling agents based on the probability distribution of the interaction information; the interaction information entropy characterizes the cooperative state determinism of each scheduling agent in the candidate adversarial scenario; and perform a weighted summation of the interaction information entropy of each scheduling agent and the cognitive entropy of each scheduling agent regarding the candidate adversarial scenario to obtain the composite entropy of each scheduling agent regarding the candidate adversarial scenario.

[0242] Each module in the training device for the aforementioned scheduling agent can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0243] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 12As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data required for training methods of scheduling agents. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a training method for scheduling agents.

[0244] Those skilled in the art will understand that Figure 12 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0245] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0246] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0247] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0248] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0249] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0250] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0251] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for training a scheduling agent, characterized in that, The method includes: Determine the parent adversarial scenario pair from the pre-set adversarial memory bank; The parent adversarial scenarios in the parent adversarial scenario pair are subjected to evolutionary processing to generate multiple candidate adversarial scenarios; By scheduling an intelligent agent to interact with the environment in each of the candidate adversarial scenarios, the cognitive entropy of the scheduling intelligent agent for each of the candidate adversarial scenarios is obtained; the cognitive entropy represents the cognitive uncertainty of the scheduling intelligent agent for the candidate adversarial scenarios. Based on the cognitive entropy of each candidate adversarial scenario, a target adversarial scenario is determined from each candidate adversarial scenario, and the internal parameters of the scheduling agent are updated based on the target adversarial scenario.

2. The method according to claim 1, characterized in that, Each adversarial scenario corresponds to a feature vector; the evolutionary processing of each parent adversarial scenario in the parent adversarial scenario pair generates multiple candidate adversarial scenarios, including: The elements in the feature vectors of each parent adversarial scenario that are in the same intersection interval are swapped to obtain the child adversarial scenario corresponding to each parent adversarial scenario; the intersection interval is determined based on two random intersection points; Add perturbation values ​​to the feature vectors of each of the sub-adversarial scenarios to generate sub-candidate adversarial scenarios for each of the sub-adversarial scenarios; The candidate adversarial scenarios for each of the aforementioned candidate adversarial scenarios are summarized to obtain the multiple candidate adversarial scenarios.

3. The method according to claim 1 or 2, characterized in that, The number of scheduling agents is one; the step of obtaining the cognitive entropy of the scheduling agent regarding each candidate adversarial scenario through environmental interaction in each candidate adversarial scenario includes: For any candidate adversarial scenario, the candidate adversarial scenario is input into the scheduling training environment, triggering the scheduling agent to interact with the candidate adversarial scenario in the environment, and obtaining the information entropy of the scheduling agent at each time step; The information entropy of the scheduling agent at each time step is aggregated to obtain the cognitive entropy of the scheduling agent regarding the candidate adversarial scenario.

4. The method according to claim 3, characterized in that, The step of obtaining the information entropy of the scheduling agent at each time step includes: For any given time step, obtain multiple state selection probabilities of the scheduling agent at that time step, and perform a logarithmic operation on each of the state selection probabilities; The logarithmic result of each state selection probability is multiplied with the corresponding state selection probability to obtain the multiple state uncertainty quantification values ​​of the scheduling agent at the time step. The information entropy of the scheduling agent at the time step is obtained by superimposing the quantified values ​​of multiple state uncertainties at the time step.

5. The method according to claim 1 or 2, characterized in that, The step of determining the target adversarial scenario from the candidate adversarial scenarios based on the cognitive entropy of each candidate adversarial scenario includes: The candidate adversarial scenarios are sorted in descending order of cognitive entropy to obtain a sequence of candidate adversarial scenarios. The target adversarial scenario is selected from the candidate adversarial scenario sequence based on the average rate of change of external reward of the scheduling agent over a preset number of historical iteration cycles.

6. The method according to claim 5, characterized in that, The step of selecting the target adversarial scenario from the candidate adversarial scenario sequence based on the average rate of change of external reward of the scheduling agent over a preset number of historical iteration cycles includes: If the average rate of change of external reward is less than a preset lower threshold, the first N candidate adversarial scenarios in the candidate adversarial scenario sequence are determined as the target adversarial scenario; N is an integer greater than 1. When the average rate of change of external reward is greater than a preset upper limit threshold, the median of the cognitive entropy of each candidate adversarial scenario is calculated, and the candidate adversarial scenario corresponding to the cognitive entropy adjacent to the median is selected from the candidate adversarial scenario sequence as the target adversarial scenario.

7. The method according to claim 1 or 2, characterized in that, The updating of the internal parameters of the scheduling agent based on the target adversarial scenario includes: The scheduling agent is trained using the target adversarial scenario to obtain the target training trajectory of the scheduling agent in the target adversarial scenario; The target training trajectory and the historical training trajectory of the scheduling agent are used as the training trajectory set; Based on the training trajectory set, the parameters of the scheduling agent are updated in batches until the convergence condition is met, thus obtaining a trained scheduling agent.

8. The method according to claim 7, characterized in that, The step of training the scheduling agent using the target adversarial scenario and obtaining the target training trajectory of the scheduling agent in the target adversarial scenario includes: The scheduling agent acquires its scheduling decisions, cognitive entropy, and environmental reward signals at each time step in the target adversarial scenario; the environmental reward signals are determined based on environmental information from the environment in which the scheduling agent executes the scheduling strategy. The cognitive entropy and environmental reward signal corresponding to each time step are weighted to obtain the combined reward signal for each time step. The scheduling decisions and combined reward information at each time step are summarized to obtain the target training trajectory of the scheduling agent in the target adversarial scenario.

9. The method according to claim 1 or 2, characterized in that, The target adversarial scenario includes multiple scenarios; the method further includes: Obtain the average cognitive entropy of each adversarial scenario in the adversarial memory bank; The cognitive entropy of each target adversarial scenario is compared with the mean cognitive entropy, and the adversarial memory is updated based on the comparison result.

10. The method according to claim 9, characterized in that, The step of updating the adversarial memory based on the comparison results includes: For any target adversarial scenario, if the cognitive entropy of the target adversarial scenario is greater than the average cognitive entropy, then the target adversarial scenario is added to the adversarial memory bank, and the adversarial scenario corresponding to the minimum cognitive entropy in the adversarial memory bank is deleted.

11. The method according to claim 1 or 2, characterized in that, The number of scheduling agents is multiple; the method further includes: For any candidate adversarial scenario, the probability distribution of the interaction information between the scheduling agents is obtained by the environmental interaction of each scheduling agent in the candidate adversarial scenario. Based on the probability distribution of the interaction information, the interaction information entropy between each of the scheduling agents is calculated; the interaction information entropy characterizes the cooperative state determinism of each of the scheduling agents in the candidate adversarial scenario. The interaction information entropy of each scheduling agent and the cognitive entropy of each scheduling agent regarding the candidate adversarial scenario are weighted and summed to obtain the composite entropy of each scheduling agent regarding the candidate adversarial scenario.

12. A training device for scheduling intelligent agents, characterized in that, The device includes: The adversarial scenario acquisition module is used to determine the parent adversarial scenario pair from the preset adversarial memory library; The adversarial scenario evolution module is used to perform evolution processing on each parent adversarial scenario in the parent adversarial scenario pair to generate multiple candidate adversarial scenarios; The cognitive entropy calculation module is used to obtain the cognitive entropy of the scheduling agent for each of the candidate adversarial scenarios by scheduling the agent to interact with the environment in each of the candidate adversarial scenarios; the cognitive entropy represents the cognitive uncertainty of the scheduling agent for the candidate adversarial scenarios. The parameter update module is used to determine the target adversarial scenario from each of the candidate adversarial scenarios based on the cognitive entropy of each candidate adversarial scenario, and update the internal parameters of the scheduling agent based on the target adversarial scenario.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 11.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 11.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 11.

Images