Algorithm Selection Method and Apparatus Based on Reinforcement Learning Algorithm Adversarial Performance

Abstract

This invention provides a method and apparatus for selecting reinforcement learning algorithms based on their adversarial effectiveness, relating to the field of reinforcement learning technology. The method includes: conducting multiple rounds of testing on multiple reinforcement learning algorithms in a target adversarial environment to obtain performance data for each algorithm during the testing process; calculating the adversarial effectiveness of each reinforcement learning algorithm based on a hierarchical evaluation index system and the performance data to determine an adversarial effectiveness score for each algorithm; and determining the algorithm selection result suitable for the target adversarial environment from among the multiple reinforcement learning algorithms based on the adversarial effectiveness score. By calculating adversarial effectiveness based on multiple indicators, a comprehensive adversarial effectiveness score is obtained. Finally, the algorithm is selected based on the scenario-specific adversarial effectiveness score, avoiding the one-sidedness caused by evaluating only a single indicator and improving the reliability of reinforcement learning algorithm selection in adversarial game scenarios.

Description

Technical Field

[0001] This invention relates to the field of reinforcement learning technology, and in particular to an algorithm selection method and apparatus based on the adversarial effectiveness of reinforcement learning algorithms. Background Technology

[0002] In multi-agent adversarial game scenarios (such as intelligent combat simulation, autonomous robot confrontation, and unmanned swarm cooperative confrontation), the effectiveness of reinforcement learning algorithms directly determines the success rate, stability, and resource utilization of task execution. With the rapid iteration of multi-agent reinforcement learning technology, its application scenarios in complex and dynamic adversarial environments are becoming increasingly widespread. However, the scientific quantitative evaluation of algorithm effectiveness and the optimal selection of scenario adaptability remain the core bottlenecks restricting the transformation of technology from theoretical research to engineering implementation.

[0003] Existing methods for algorithm selection in multi-agent adversarial game scenarios generally use the adversarial win rate as the core evaluation indicator to evaluate and select different algorithms. Evaluating algorithms based on a single indicator leads to low reliability of algorithm selection. Summary of the Invention

[0004] This invention provides an algorithm selection method and apparatus based on the adversarial effectiveness of reinforcement learning algorithms, which improves the reliability of algorithm selection in multi-agent adversarial game scenarios.

[0005] This invention provides an algorithm selection method based on the adversarial effectiveness of reinforcement learning algorithms, comprising the following steps: Multiple reinforcement learning algorithms are tested in a target adversarial environment for multiple rounds to obtain performance data of each algorithm during the testing process; the testing includes training testing and adversarial testing. Based on the hierarchical evaluation index system and the performance data, the adversarial effectiveness of each reinforcement learning algorithm is calculated to determine the adversarial effectiveness score of each reinforcement learning algorithm. The hierarchical evaluation index system is constructed based on the environmental simulation parameters of the target adversarial environment and the algorithm training parameters of each reinforcement learning algorithm. The hierarchical evaluation index system includes time response and convergence performance index, reward function performance index, and adversarial performance index. The time response and convergence performance index is used to characterize the response time characteristics of the algorithm's interaction with the environment and the convergence characteristics of the loss function during training. The reward function performance index is used to characterize the reward accumulation characteristics and reward fluctuation characteristics of the algorithm during adversarial process. The adversarial performance index is used to characterize the adversarial combat capability of the algorithm. Based on the adversarial effectiveness score, an algorithm selection result suitable for the target adversarial environment is determined from the plurality of reinforcement learning algorithms.

[0006] According to the present invention, an algorithm selection method based on the adversarial effectiveness of reinforcement learning algorithms is provided. The method involves calculating the adversarial effectiveness of each reinforcement learning algorithm based on a hierarchical evaluation index system and the performance data, and determining the adversarial effectiveness score of each algorithm. This includes: According to the hierarchical evaluation index system, the performance data of each reinforcement learning algorithm are calculated to determine the initial index data of each reinforcement learning algorithm; Based on the architecture type information of each reinforcement learning algorithm, the indicator items that are not applicable to each reinforcement learning algorithm in the hierarchical evaluation indicator system are determined, and the inapplicable indicator items are assigned a preset benchmark value to obtain the adaptive indicator data. The comprehensive weight of each indicator in the hierarchical evaluation index system is determined, and the adaptation index data is weighted based on the comprehensive weight to obtain the adversarial effectiveness score of each reinforcement learning algorithm.

[0007] According to the algorithm selection method based on the adversarial effectiveness of reinforcement learning algorithms provided by the present invention, the step of determining the comprehensive weight of each indicator in the hierarchical evaluation index system includes: Based on the judgment matrix constructed using the preset scaling method, the feature vectors of each indicator are determined, and the feature vectors are normalized to obtain the first weight of each indicator. Based on the performance data of each reinforcement learning algorithm on various indicators, the data information entropy of each indicator is determined, and the second weight of each indicator is determined based on the data information entropy. The first weight and the second weight are linearly weighted and fused to obtain the comprehensive weight of each indicator.

[0008] According to the present invention, an algorithm selection method based on the adversarial effectiveness of reinforcement learning algorithms is provided, wherein the time response and convergence performance indicators include the average response time of the environment simulation, the variance of the algorithm execution response time, the value function loss, and the maximum policy loss. The performance metrics of the reward function include cumulative reward data, average reward data, and reward fluctuation data. The performance indicators for combat include win rate in multiple rounds of combat, enemy destruction rate, and our own loss rate.

[0009] According to the present invention, an algorithm selection method based on the adversarial performance of reinforcement learning algorithms is provided, wherein the multi-round adversarial win rate is determined based on the number of wins and losses of each reinforcement learning algorithm in the multi-round adversarial test; The enemy destruction rate is determined based on the ratio of the number of enemies destroyed in each round of multi-round combat testing to the initial number of enemies; The stated loss rate is determined based on the ratio of the number of losses suffered by our side in each round of multi-round adversarial testing to our initial number.

[0010] According to the algorithm selection method based on the adversarial effectiveness of reinforcement learning algorithms provided by the present invention, the cumulative reward data is determined by statistically analyzing the instantaneous rewards of each reinforcement learning algorithm according to the test time step. The average reward data is determined by dividing the cumulative reward by the total number of time steps in a single test cycle; The reward fluctuation data is determined based on the variance of each instant reward relative to the average reward data within a single test cycle.

[0011] The present invention also provides an algorithm selection device based on the adversarial effectiveness of reinforcement learning algorithms, comprising the following modules: The testing module is used to perform multiple rounds of testing on various reinforcement learning algorithms in a target adversarial environment, and to obtain performance data of each reinforcement learning algorithm during the testing process; the testing includes training testing and adversarial testing. The performance calculation module is used to calculate the adversarial performance of each reinforcement learning algorithm based on the hierarchical evaluation index system and the performance data, and to determine the adversarial performance score of each reinforcement learning algorithm. The hierarchical evaluation index system is constructed based on the environmental simulation parameters of the target adversarial environment and the algorithm training parameters of each reinforcement learning algorithm. The hierarchical evaluation index system includes time response and convergence performance index, reward function performance index, and adversarial performance index. The time response and convergence performance index is used to characterize the response time characteristics of the algorithm's interaction with the environment and the convergence characteristics of the loss function during training. The reward function performance index is used to characterize the reward accumulation characteristics and reward fluctuation characteristics of the algorithm during adversarial process. The adversarial performance index is used to characterize the adversarial combat capability of the algorithm. The selection module is used to determine the algorithm selection result that is suitable for the target adversarial environment from the plurality of reinforcement learning algorithms based on the adversarial effectiveness score.

[0012] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the algorithm selection method based on the adversarial effectiveness of reinforcement learning algorithms as described above.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the algorithm selection method based on the adversarial effectiveness of reinforcement learning algorithms as described above.

[0014] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the algorithm selection method based on the adversarial effectiveness of reinforcement learning algorithms as described above.

[0015] This invention provides a method and apparatus for selecting reinforcement learning algorithms based on their adversarial performance. By conducting multiple rounds of testing, including training and adversarial tests, on multiple reinforcement learning algorithms in a target adversarial environment, comprehensive performance data is obtained. Then, based on a hierarchical evaluation index system covering time response and convergence performance indicators, reward function performance indicators, and adversarial performance indicators, the adversarial performance of each reinforcement learning algorithm is calculated, resulting in a comprehensive adversarial performance score. Finally, the algorithm is selected based on this adversarial performance score in a scenario-based manner. This comprehensively characterizes the overall performance of reinforcement learning algorithms in adversarial scenarios from multiple dimensions, including time response, training convergence, reward acquisition, and actual adversarial performance, avoiding the one-sidedness of evaluating based on a single index and improving the reliability of reinforcement learning algorithm selection in adversarial game scenarios. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the algorithm selection method based on the adversarial effectiveness of reinforcement learning algorithms provided by the present invention.

[0018] Figure 2 This is a schematic diagram of the algorithm evaluation initialization provided by the present invention.

[0019] Figure 3 This is a schematic diagram of the hierarchical evaluation index system structure provided by the present invention.

[0020] Figure 4 This is a flowchart illustrating the algorithm selection method based on the adversarial effectiveness of reinforcement learning algorithms provided by this invention.

[0021] Figure 5 This is a schematic diagram of the algorithm selection device based on the adversarial effectiveness of reinforcement learning algorithms provided by the present invention.

[0022] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0024] Figure 1 This is a flowchart illustrating the algorithm selection method based on the adversarial effectiveness of reinforcement learning algorithms provided by this invention, as shown below. Figure 1 As shown, the method includes the following: Step 110: Perform multiple rounds of testing on multiple reinforcement learning algorithms in a target adversarial environment to obtain performance data of each reinforcement learning algorithm during the testing process; the testing includes training testing and adversarial testing; Step 120: Based on the hierarchical evaluation index system and the performance data, calculate the adversarial effectiveness of each reinforcement learning algorithm and determine the adversarial effectiveness score of each reinforcement learning algorithm. The hierarchical evaluation index system is constructed based on the environmental simulation parameters of the target adversarial environment and the algorithm training parameters of each reinforcement learning algorithm. The hierarchical evaluation index system includes time response and convergence performance index, reward function performance index, and adversarial performance index. The time response and convergence performance index is used to characterize the response time characteristics of the algorithm's interaction with the environment and the convergence characteristics of the loss function during training. The reward function performance index is used to characterize the reward accumulation characteristics and reward fluctuation characteristics of the algorithm during adversarial process. The adversarial performance index is used to characterize the adversarial combat capability of the algorithm. Step 130: Based on the adversarial effectiveness score, determine the algorithm selection result that is suitable for the target adversarial environment from the multiple reinforcement learning algorithms.

[0025] The execution subject of the algorithm selection method based on the adversarial effectiveness of reinforcement learning algorithms provided by this invention can be an electronic device, a component in an electronic device, an integrated circuit, or a chip. The electronic device can be a mobile electronic device or a non-mobile electronic device. For example, a mobile electronic device can be a mobile phone, tablet computer, laptop computer, PDA, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc., while a non-mobile electronic device can be a server, network attached storage (NAS), or personal computer (PC), etc. This invention does not impose specific limitations.

[0026] The following example, using a computer executing the algorithm selection method based on the adversarial effectiveness of reinforcement learning algorithms provided by this invention, illustrates the technical solution of this invention in detail.

[0027] In step 110, multiple rounds of testing are conducted on various reinforcement learning algorithms in a target adversarial environment to obtain performance data of each algorithm during the testing process.

[0028] A target adversarial environment refers to a real-world adversarial game scenario that requires the deployment of reinforcement learning algorithms. A target adversarial environment can be any type of multi-agent adversarial game scenario, such as intelligent combat simulation scenarios, autonomous robot adversarial scenarios, or unmanned swarm cooperative adversarial scenarios. A target adversarial environment typically involves two adversarial parties, each controlling a certain number of agents, performing adversarial tasks within a simulated environment.

[0029] In this invention, "multiple reinforcement learning algorithms" refers to a set of candidate algorithms for performance evaluation and selection. These multiple reinforcement learning algorithms can include multi-agent reinforcement learning algorithms of different types and architectures. For example, they may include QMIX (a value-based multi-agent reinforcement learning algorithm), MADDPG (multi-agent deep deterministic policy gradient algorithm), and MAPPO (multi-agent proximal policy optimization algorithm), etc.

[0030] Testing multiple reinforcement learning algorithms in multiple rounds refers to repeatedly testing each candidate reinforcement learning algorithm in a unified adversarial environment to collect statistically significant performance data.

[0031] The number of rounds in the multi-round testing can be set according to actual needs, for example, up to 30 rounds, to ensure that the collected performance data has sufficient statistical reliability. During the multi-round testing, all reinforcement learning algorithms are tested under the same objective adversarial environment configuration to ensure the fairness of the evaluation.

[0032] The testing includes training testing and adversarial testing. Training testing refers to testing the reinforcement learning algorithm during training in a target adversarial environment. It collects performance data related to the algorithm's training characteristics, such as the algorithm's environmental interaction response time, changes in the loss function during training, and the stability of the algorithm's execution time. Adversarial testing refers to testing the trained reinforcement learning algorithm in a target adversarial environment using actual adversarial tasks. It collects performance data related to the algorithm's real-world performance, such as win rate, kill rate, loss rate, and reward acquisition. By dividing the testing process into training and adversarial testing phases, comprehensive performance data of the reinforcement learning algorithm during both the training and real-world phases can be obtained.

[0033] Performance data refers to quantifiable performance metrics generated by various reinforcement learning algorithms during training, testing, and adversarial testing. Performance data may include, but is not limited to: environment response time data, algorithm execution time data, loss function values, reward data, win / loss results data, number of kills, and number of friendly losses. Specifically, performance data can be collected by setting up a data collection interface in the target adversarial environment to synchronously collect raw data for various performance metrics during the test run.

[0034] In step 120, based on the hierarchical evaluation index system and the performance data, the adversarial effectiveness of each reinforcement learning algorithm is calculated to determine the adversarial effectiveness score of each reinforcement learning algorithm.

[0035] A hierarchical evaluation index system is pre-constructed. This system is a structured index system used to evaluate the performance data of various reinforcement learning algorithms from multiple dimensions. The hierarchical evaluation index system is constructed based on the environmental simulation parameters of the adversarial environment and the algorithm training parameters of each reinforcement learning algorithm. For example... Figure 2 The algorithm evaluation initialization diagram provided by this invention illustrates that environmental simulation parameters and algorithm training parameters can be weighted and initialized based on an initial weight vector. Environmental simulation parameters refer to the basic configuration parameters for operating in the target adversarial environment, such as simulation step size and single-round simulation duration. Algorithm training parameters refer to the configuration parameters used by each reinforcement learning algorithm during training, such as the maximum number of training steps and performance data sampling frequency. Based on the environmental simulation parameters and algorithm training parameters, the calculation benchmarks and data collection specifications for each indicator in the evaluation index system can be determined.

[0036] The hierarchical evaluation index system includes three primary indexes: time response and convergence performance index, reward function performance index, and adversarial performance index. The time response and convergence performance index characterizes the response time characteristics of the algorithm's interaction with the environment and the convergence characteristics of the loss function during training. Response time characteristics reflect the real-time performance and stability of the algorithm during interaction with the adversarial environment, while the convergence characteristics of the loss function reflect the accuracy and stability of model parameter optimization during training. The reward function performance index characterizes the algorithm's reward accumulation and reward fluctuation characteristics during adversarial processes. Reward accumulation characteristics reflect the overall effectiveness and average level of reward acquisition by the algorithm in adversarial tasks, while reward fluctuation characteristics reflect the stability of reward acquisition. The adversarial performance index characterizes the algorithm's adversarial combat capability, directly reflecting core practical indicators such as win rate, strike effectiveness, and self-defense capability in actual adversarial scenarios.

[0037] The adversarial performance score is a quantitative score obtained by comprehensively calculating the performance data of each reinforcement learning algorithm based on a hierarchical evaluation index system. The adversarial performance score can be a value within a preset range (e.g., between 0 and 1), with a higher value indicating better overall adversarial performance of the corresponding reinforcement learning algorithm.

[0038] In step 130, the algorithm selection result that is suitable for the target adversarial environment is determined from multiple reinforcement learning algorithms based on the adversarial effectiveness score.

[0039] For example, the reinforcement learning algorithm with the highest adversarial effectiveness score can be used as the algorithm selection result, that is, the algorithm with the best overall performance can be recommended as the first choice in a target adversarial environment. Alternatively, multiple reinforcement learning algorithms can be ranked from high to low according to their adversarial effectiveness scores, and the ranking results can be output for decision-makers to refer to. Furthermore, by combining the adversarial effectiveness score and the performance of individual indicators, differentiated algorithm selection results can be given for different scenario requirements, such as prioritizing real-time performance or stability.

[0040] This invention provides an algorithm selection method based on the adversarial performance of reinforcement learning algorithms. By conducting multiple rounds of testing, including training and adversarial tests, on multiple reinforcement learning algorithms in a target adversarial environment, comprehensive performance data is obtained. Then, adversarial performance is calculated for each reinforcement learning algorithm based on a hierarchical evaluation index system covering time response and convergence performance indicators, reward function performance indicators, and adversarial performance indicators, resulting in a comprehensive adversarial performance score. Finally, the algorithm is selected based on the scenario-specific adversarial performance score. This method comprehensively characterizes the overall performance of reinforcement learning algorithms in adversarial scenarios from multiple dimensions, including time response, training convergence, reward acquisition, and actual adversarial performance, avoiding the one-sidedness of evaluating based on a single index and improving the reliability of reinforcement learning algorithm selection in adversarial game scenarios.

[0041] In one embodiment, the adversarial effectiveness calculation of each reinforcement learning algorithm based on the hierarchical evaluation index system and the performance data, and the determination of the adversarial effectiveness score of each reinforcement learning algorithm, includes: According to the hierarchical evaluation index system, the performance data of each reinforcement learning algorithm are calculated to determine the initial index data of each reinforcement learning algorithm; Based on the architecture type information of each reinforcement learning algorithm, the indicator items that are not applicable to each reinforcement learning algorithm in the hierarchical evaluation indicator system are determined, and the inapplicable indicator items are assigned a preset benchmark value to obtain the adaptive indicator data. The comprehensive weight of each indicator in the hierarchical evaluation index system is determined, and the adaptation index data is weighted based on the comprehensive weight to obtain the adversarial effectiveness score of each reinforcement learning algorithm.

[0042] According to the hierarchical evaluation index system, the performance data of each reinforcement learning algorithm are calculated to determine the initial index data of each reinforcement learning algorithm.

[0043] Initial performance data refers to the numerical values ​​of various indicators obtained by statistically calculating the raw performance data collected during the testing process of each reinforcement learning algorithm, according to the definitions and calculation rules of each indicator in the hierarchical evaluation indicator system. For example, the average response time of the environment simulation can be calculated based on the environment response time data collected in multiple rounds of testing; the variance of the algorithm execution response time can be calculated based on the algorithm execution time data; and the value function loss or policy loss can be calculated based on the loss function value data during training. The calculation process of initial performance data is strictly carried out according to the quantitative formulas of each indicator in the hierarchical evaluation indicator system to ensure consistency in the indicator calculation methods among different reinforcement learning algorithms.

[0044] Based on the architecture type information of each reinforcement learning algorithm, the inapplicable index items in the hierarchical evaluation index system are identified, and the inapplicable index items are assigned preset benchmark values ​​to obtain the adapted index data.

[0045] Architecture type information refers to the category of technical architecture used by each reinforcement learning algorithm. Different types of reinforcement learning algorithms may have some inapplicable metrics in a hierarchical evaluation system due to their different underlying architectures. For example, reinforcement learning algorithms based on value decomposition architectures (such as QMIX) have a value function loss metric during training, while reinforcement learning algorithms based on policy gradient frameworks (such as MADDPG and MAPPO) do not. Conversely, algorithms of different architecture types may all have policy loss metrics, but certain specific architectures may lack certain other specific metrics.

[0046] Assigning inapplicable metrics to preset benchmark values ​​is a strategy for adapting to differences in algorithm characteristics. The preset benchmark value can be set according to actual needs, for example, it can be set to 0. By assigning inapplicable metrics to preset benchmark values ​​(such as 0), the differences in metrics between different types of reinforcement learning algorithms can be adapted without changing the evaluation system architecture, enabling horizontal comparisons between different types of algorithms within a unified evaluation framework.

[0047] For example, for the QMIX algorithm, since it uses a value decomposition architecture, it has a value function loss metric, but the policy loss metric may not be applicable; therefore, its policy loss metric can be set to 0. For the MADDPG and MAPPO algorithms, since they use a policy gradient framework, they do not have a value function loss metric; therefore, their value function loss metric can be set to 0. This strategy of "filling missing metrics with 0" establishes a fair basis for comparison between different types of algorithms.

[0048] The comprehensive weight of each indicator in the hierarchical evaluation index system is determined, and the adaptive index data is weighted based on the comprehensive weight to obtain the adversarial effectiveness score of each reinforcement learning algorithm.

[0049] The process of determining the comprehensive weight of each indicator in the hierarchical evaluation index system and weighting the matching indicator data based on the comprehensive weight to obtain the adversarial effectiveness score can be as follows: normalize the matching indicator data of each indicator and then sum them with the corresponding comprehensive weight to obtain the comprehensive score.

[0050] Optionally, before performing weighted summation, the data for the applicable indicators can be normalized to eliminate differences in units and numerical ranges between different indicators. Normalization can be performed in different ways depending on the type of indicator: for positive indicators (where a larger value indicates better performance), a positive normalization formula can be used; for negative indicators (where a smaller value indicates better performance), a negative normalization formula can be used.

[0051] Alternatively, the normalization formula for the positive index can be: The normalization formula for negative indices can be: .in, The original value, and These are the minimum and maximum values ​​of this indicator across all algorithms, respectively.

[0052] The formula for calculating the combat effectiveness score can be: ,in, The adversarial performance score for the i-th reinforcement learning algorithm is given. The comprehensive weight of the j-th indicator is... This represents the normalized adaptation index data for the i-th reinforcement learning algorithm on the j-th metric. The adversarial effectiveness score can range from 0 to 1, with a higher score indicating better overall adversarial effectiveness for the corresponding reinforcement learning algorithm.

[0053] In one embodiment, determining the comprehensive weight of each indicator in the hierarchical evaluation index system includes: Based on the judgment matrix constructed using the preset scaling method, the feature vectors of each indicator are determined, and the feature vectors are normalized to obtain the first weight of each indicator. Based on the performance data of each reinforcement learning algorithm on various indicators, the data information entropy of each indicator is determined, and the second weight of each indicator is determined based on the data information entropy. The first weight and the second weight are linearly weighted and fused to obtain the comprehensive weight of each indicator.

[0054] Based on the judgment matrix constructed using the preset scaling method, the feature vectors of each indicator are determined, and the feature vectors are normalized to obtain the first weight of each indicator.

[0055] The first weight is determined based on subjective judgment, reflecting the domain experts' experiential assessment of the importance of each indicator. Preset scaling methods can employ scaling techniques commonly used in the Analytic Hierarchy Process (AHP), such as the 1-9 scaling method. The 1-9 scaling method uses integers from 1 to 9 and their reciprocals to represent the relative importance between two indicators, where 1 indicates that both indicators are equally important, and 9 indicates that one indicator is extremely important relative to the other.

[0056] First, construct a hierarchical structure, including the target layer (comprehensive algorithm performance evaluation), the criteria layer (three types of primary indicators), and the solution layer (various secondary indicators). Then, invite domain experts (e.g., 5 to 8 experts) to make pairwise comparisons of the relative importance of each indicator, and construct a judgment matrix based on the judgment results.

[0057] Based on the constructed judgment matrix, the eigenvectors of the judgment matrix are calculated, and the eigenvectors are normalized to obtain the first weights of each indicator. It should be noted that before obtaining the first weights, a consistency check needs to be performed on the judgment matrix to verify the logical consistency of the expert judgments. The consistency check can be achieved by calculating the consistency ratio (CR). When the consistency ratio meets a preset condition (e.g., CR < 0.1), the judgment matrix is ​​considered to have satisfactory consistency, and the obtained first weights are acceptable.

[0058] Based on the performance data of each reinforcement learning algorithm on various indicators, the data information entropy of each indicator is determined, and the second weight of each indicator is determined based on the data information entropy.

[0059] In this invention, the second weight is determined based on objective data characteristics, reflecting the amount of information carried by each indicator's data. The second weight can be calculated using the entropy weight method. The basic principle of the entropy weight method is: if the data difference of a certain indicator among all algorithms is greater, then the indicator contains more information, and its contribution to distinguishing the performance of different algorithms is also greater, thus it should be assigned a higher weight.

[0060] The process of calculating data information entropy may include: first, analyzing the performance data of each reinforcement learning algorithm on various metrics. Normalization is performed to obtain the normalized matrix. Where m is the number of reinforcement learning algorithms to be evaluated and n is the number of indicators; then the information entropy of each indicator is calculated. The smaller the information entropy, the greater the data dispersion of the indicator and the more information it contains; finally, the second weight of each indicator is calculated based on the information entropy. The smaller the information entropy, the greater the second weight is obtained.

[0061] Optionally, the formula for calculating information entropy can be: Where k is a constant (usually taken as k) The formula for calculating the second weight can be: .

[0062] Finally, the first weight and the second weight are linearly weighted and merged to obtain the comprehensive weight of each indicator.

[0063] In this invention, linear weighted fusion refers to linearly combining the first weight (subjective weight) and the second weight (objective weight) according to a preset fusion coefficient to obtain a comprehensive weight. The fusion coefficient can be set according to actual needs. For example, a subjective coefficient α and an objective coefficient (1-α) can be set, and the formula for calculating the comprehensive weight is: ,in The first weight of the j-th indicator is... , which is the second weight of the j-th indicator.

[0064] Optionally, the subjective coefficient α can be set to 0.6, and the objective coefficient (1-α) can be set to 0.4, meaning that the subjective weight accounts for 60% and the objective weight accounts for 40% in the overall weighting. This setting ensures that the experience of domain experts guides the evaluation direction while incorporating objective data characteristics for correction, thus balancing the professionalism and objectivity of the evaluation. It should be noted that the specific value of the fusion coefficient can be flexibly adjusted according to the characteristics of the application scenario. For example, in scenarios with a high degree of reliance on data, the proportion of the objective coefficient can be appropriately increased.

[0065] In one embodiment, the time response and convergence performance metrics include the average response time of the environmental simulation, the variance of the algorithm execution response time, the value function loss, and the maximum policy loss. The performance metrics of the reward function include cumulative reward data, average reward data, and reward fluctuation data. The performance indicators for combat include win rate in multiple rounds of combat, enemy destruction rate, and our own loss rate.

[0066] The indicator structure in a hierarchical evaluation indicator system can be as follows: Figure 3The hierarchical evaluation index system structure provided by this invention is illustrated in the diagram. The average response time of the environment simulation refers to the average time it takes for the environment to return a response when each reinforcement learning algorithm interacts with the target adversarial environment in multiple simulation runs. This index reflects the real-time performance of the algorithm's interaction with the environment; the smaller the index value, the stronger the real-time performance of the environment interaction. The average response time of the environment simulation can be calculated based on the arithmetic mean of the response times of each environment in multiple simulations. Optionally, the formula for calculating the average response time of the environment simulation can be: ,in Let be the environmental response time for the i-th simulation (in milliseconds), and M be the number of simulations (M can be set to be greater than or equal to 30 to ensure statistical significance). The average response time of the environmental simulation is a negative indicator.

[0067] Algorithm execution response time variance refers to the variance of the execution time of each reinforcement learning algorithm relative to the mean execution time during multiple executions. This metric reflects the stability of the algorithm; a smaller value indicates stronger stability. The variance of algorithm execution response time can be calculated based on the average of the sum of squared deviations of multiple execution times from the mean execution time. Optionally, the formula for calculating the variance of algorithm execution response time can be: ,in Let be the execution time of the i-th algorithm. This represents the mean execution time. The variance of the algorithm's execution response time is a negative indicator.

[0068] Value function loss refers to the value function loss value of a reinforcement learning algorithm based on a value decomposition architecture at the end of training. This metric reflects the accuracy of the algorithm's value estimation; a smaller value indicates higher accuracy. It should be noted that value function loss is only applicable to reinforcement learning algorithms based on a value decomposition architecture (such as the QMIX algorithm). For reinforcement learning algorithms based on a policy gradient framework (such as the MADDPG and MAPPO algorithms), since their architecture does not involve value function decomposition, this metric is not applicable. In such cases, the value function loss can be assigned a preset baseline value according to the aforementioned algorithm characteristics adaptation strategy. Optionally, the value function loss can be determined based on the average value function loss of the last few training steps, for example: ,in Let H be the value function loss at step k, and H be the number of steps in the last few steps. The value function loss is a negative index.

[0069] Maximum policy loss refers to the maximum policy loss of each reinforcement learning algorithm during training. This metric reflects the stability of the algorithm's policy updates; a smaller value indicates less fluctuation and stronger stability during policy updates. Maximum policy loss is applicable to all types of reinforcement learning algorithms. Optionally, the formula for calculating maximum policy loss can be: ,in This represents the policy loss at step k. The maximum policy loss is a negative indicator.

[0070] The performance metrics for the reward function include cumulative reward data, average reward data, and reward volatility data.

[0071] Cumulative reward data refers to the sum of all immediate rewards obtained by each reinforcement learning algorithm within a single round of adversarial testing. This metric reflects the overall effectiveness of the algorithm in achieving its objective in adversarial tasks; a higher value indicates better objective achievement. Cumulative reward data is strongly correlated with the task objective, reflecting the algorithm's comprehensive reward acquisition capability throughout the adversarial process. Cumulative reward data is a positive metric.

[0072] Average reward data refers to the average reward per step obtained by dividing the cumulative reward by the total number of time steps within a single test cycle. This metric eliminates the influence of simulation duration on the total reward, facilitating comparisons across multiple scenarios with different simulation duration configurations. Average reward data is a positive metric.

[0073] Reward volatility data refers to the dispersion of each reinforcement learning algorithm's immediate reward relative to the average reward within a single test cycle. This metric reflects the stability of the algorithm's reward acquisition; a smaller value indicates more stable reward acquisition, meaning the algorithm's performance is more consistent during adversarial processes. Reward volatility data is a negative metric.

[0074] The performance indicators for combat include the win rate in multiple rounds of combat, the kill rate against the enemy, and the loss rate of our side.

[0075] Multi-round adversarial win rate refers to the probability that each reinforcement learning algorithm wins in a multi-round adversarial test. This metric directly reflects the algorithm's overall adversarial advantage in adversarial scenarios; a higher metric value indicates a more significant adversarial advantage. Multi-round adversarial win rate is a positive indicator.

[0076] The kill rate refers to the average ratio of the number of enemy agents destroyed by each reinforcement learning algorithm in multiple rounds of adversarial testing to the initial number of enemy agents. This metric reflects the algorithm's effectiveness; a higher value indicates stronger effectiveness. The kill rate is a positive indicator.

[0077] Our loss rate refers to the average ratio of the number of our agents lost to the initial number of our agents in multi-round adversarial testing for each reinforcement learning algorithm. This metric reflects the algorithm's resource protection capability; a smaller value indicates better resource protection. Our loss rate is a negative metric.

[0078] In one embodiment, the specific calculation methods for multi-round combat win rate, enemy destruction rate, and friendly loss rate are as follows: The multi-round adversarial win rate is determined based on the number of wins and losses for each reinforcement learning algorithm in multi-round adversarial testing. Specifically, the multi-round adversarial win rate equals the number of wins divided by the sum of the number of wins and losses. Optionally, the formula for calculating the multi-round adversarial win rate can be: Where W represents the number of wins and L represents the number of losses. It should be noted that if a tie occurs during the test, the result of that round is not counted in the win / loss statistics; a retest can be conducted to obtain a definitive result.

[0079] The enemy kill rate is determined based on the ratio of the number of enemy units destroyed in each round of multi-round combat testing to the initial number of enemy units. Specifically, the enemy kill rate is equal to the average of the ratios of the number of enemy units destroyed in each round to the initial number of enemy units across all rounds. Optionally, the formula for calculating the enemy kill rate can be: D Where K is the total number of rounds, The number of enemy units destroyed in round k. denoted as the initial number of enemies in round k. The enemy destruction rate quantitatively measures the algorithm's effectiveness against the enemy. Unlike a simple win rate metric, the enemy destruction rate can distinguish between narrow victories and decisive wins, providing a more granular measure for evaluating the algorithm's effectiveness.

[0080] Our loss rate is determined based on the ratio of our losses in each round to our initial number in a multi-round adversarial test. Specifically, our loss rate is equal to the average of the ratios of our losses in each round to our initial number across all rounds. Optionally, the formula for calculating our loss rate can be: Where K is the total number of rounds, Let K be the number of losses suffered by our side in round k. Let be the initial number of our side in round k. Our loss rate reflects the algorithm's ability to protect its own resources during the confrontation. A lower our loss rate means that the algorithm can not only effectively complete the attack mission, but also better preserve its own strength during the confrontation.

[0081] Through the above specific calculation methods, the three indicators in the adversarial effectiveness index quantify the actual adversarial combat capability of the algorithm from three dimensions: win rate, strike effectiveness, and resource protection. They complement and corroborate each other, and can comprehensively and accurately reflect the algorithm's actual combat performance in adversarial scenarios.

[0082] In one embodiment, the cumulative reward data, average reward data, and reward fluctuation data are calculated as follows: The cumulative reward data is determined by calculating the immediate rewards of each reinforcement learning algorithm over each test time step. Specifically, the cumulative reward data equals the sum of the immediate rewards at all time steps within a single test cycle. Optionally, the formula for calculating the cumulative reward data can be: ,in Let t be the immediate reward at time t. The immediate reward is the reward value given by the adversarial environment at each time step based on the behavioral performance of the agents controlled by each reinforcement learning algorithm. The immediate reward is strongly correlated with the task objective. For example, in a red-blue adversarial scenario, the rules for the immediate reward can be set as follows: destroying the enemy agent grants a positive reward (e.g., +10 points), protecting the friendly agent grants a positive reward (e.g., +5 points), the friendly agent being destroyed grants a negative reward (e.g., -8 points), and a tie due to timeout grants zero reward, etc.

[0083] The average reward is determined by dividing the cumulative reward by the total number of time steps in a single test cycle. Specifically, the average reward equals the cumulative reward divided by the total number of time steps in a single test cycle. Optionally, the formula for calculating the average reward can be: , where T is the total number of time steps within a single test cycle. Average reward data eliminates the impact of simulation duration on the total reward, enabling fair comparisons between algorithms running under different simulation duration configurations and improving the cross-scenario applicability of the metrics.

[0084] Reward volatility data is determined based on the variance of each instant reward relative to the average reward within a single testing cycle. Specifically, reward volatility data equals the variance of the difference between the instant reward and the average reward at each time step within a single testing cycle. Optionally, the formula for calculating reward volatility data can be: Reward fluctuation data quantifies the stability of an algorithm's reward acquisition during adversarial processes. Smaller reward fluctuation data indicates that the algorithm's performance is more stable and consistent, without sudden highs and lows, and is of great reference value in adversarial tasks that require stable and continuous achievement of objectives.

[0085] Optionally, before implementing the algorithm selection method based on the adversarial effectiveness of reinforcement learning algorithms, evaluation parameters can be initialized to set the basic parameters of the adversarial scenario and the initial parameters of the evaluation system, providing a unified benchmark for subsequent indicator collection and calculation.

[0086] Evaluation parameter initialization includes setting environmental simulation parameters, algorithm training parameters, and initial weight vectors.

[0087] Environmental simulation parameters can include: simulation step size and single-round simulation duration. The simulation step size is the temporal resolution of the target adversarial environment during simulation. For example, a simulation step size Δt = 0.1 seconds can be set to meet the time accuracy requirements of real-time adversarial scenarios and avoid excessively large simulation step sizes that could lead to latency affecting the adversarial results. The single-round simulation duration is the duration of each round of adversarial testing. For example, a single-round simulation duration T = 300 seconds can be set to cover the entire adversarial cycle and ensure sufficient collection of performance data.

[0088] Algorithm training parameters may include: maximum number of training steps and performance data sampling frequency. The maximum number of training steps is the maximum number of training iterations each reinforcement learning algorithm can undergo during training and testing. For example, a maximum training step count of N = 10,000 steps can be set to ensure the algorithm trains to a convergent and stable state. The performance data sampling frequency is the frequency at which performance data is collected during testing. For example, a performance data sampling frequency of f = 10 Hz can be set to balance data precision and storage redundancy.

[0089] The initial weight vector is the initial weight allocation for the three primary indicators in the hierarchical evaluation index system, represented as W = [w1, w2, w3], where w1 is the weight proportion of the time response and convergence efficiency indicators, w2 is the weight proportion of the reward function efficiency indicator, and w3 is the weight proportion of the adversarial efficiency indicator. The initial weight vector can be used as the starting value for subsequent combined weight method optimization.

[0090] During the data acquisition phase, each reinforcement learning algorithm can be run in a standardized adversarial environment for multiple rounds of testing (e.g., 30 rounds each). The raw performance data of each indicator in the hierarchical evaluation index system can be collected simultaneously, as shown in Table 1, the raw data table of algorithm performance indicators.

[0091] Secondary indicators QMIX MADDPG MAPPO Indicator Type Average response time of environmental simulation (ms) 45 52 38 negative Algorithm execution response time variance 12.5 18.3 9.7 negative Value function final loss 0.021 0 0 negative Maximum strategy loss 0.035 0.028 0.022 negative Multi-agent cumulative reward 1250 1420 1680 positive Average reward 4.17 4.73 5.60 positive Reward fluctuation 1.8 2.3 1.5 negative Multi-round match win rate 0.65 0.72 0.85 positive Enemy kill rate 0.62 0.68 0.78 positive Our loss rate 0.35 0.28 0.22 negative This invention also provides a flowchart illustrating the algorithm selection method based on the adversarial effectiveness of reinforcement learning algorithms provided by this invention, as shown below. Figure 4 The flowchart illustrating the algorithm selection method based on the adversarial effectiveness of reinforcement learning algorithms provided by this invention is shown below. Specifically, it includes: First, initializing a hierarchical evaluation index system. Based on this, subjective weights and objective weights are obtained separately, and combined to obtain a weight combination. Then, the system collects performance data and performs index normalization on this data. After processing, adjustments are made to address differences in algorithm characteristics, specifically by padding missing values ​​with zeros. Next, the adversarial effectiveness score is calculated based on the above processing results, and this score is used to evaluate and compare the performance of each algorithm. Finally, the optimal algorithm is output based on the comparison results, and the entire process concludes.

[0092] The following describes the algorithm selection device based on the adversarial performance of reinforcement learning algorithms provided by the present invention. The algorithm selection device based on the adversarial performance of reinforcement learning algorithms described below and the algorithm selection method based on the adversarial performance of reinforcement learning algorithms described above can be referred to in correspondence with each other.

[0093] like Figure 5 A schematic diagram of the structure of the algorithm selection device based on the adversarial effectiveness of reinforcement learning algorithms provided by the present invention, the device comprising: Test module 510 is used to perform multiple rounds of testing on multiple reinforcement learning algorithms in a target adversarial environment and obtain performance data of each reinforcement learning algorithm during the testing process; the testing includes training testing and adversarial testing; The performance calculation module 520 is used to calculate the adversarial performance of each reinforcement learning algorithm based on the hierarchical evaluation index system and the performance data, and to determine the adversarial performance score of each reinforcement learning algorithm. The hierarchical evaluation index system is constructed based on the environmental simulation parameters of the target adversarial environment and the algorithm training parameters of each reinforcement learning algorithm. The hierarchical evaluation index system includes time response and convergence performance index, reward function performance index, and adversarial performance index. The time response and convergence performance index is used to characterize the response time characteristics of the algorithm's interaction with the environment and the convergence characteristics of the loss function during training. The reward function performance index is used to characterize the reward accumulation characteristics and reward fluctuation characteristics of the algorithm during adversarial process. The adversarial performance index is used to characterize the adversarial combat capability of the algorithm. Selection module 530 is used to determine the algorithm selection result that is suitable for the target adversarial environment from the plurality of reinforcement learning algorithms based on the adversarial effectiveness score.

[0094] This invention provides an algorithm selection device based on the adversarial performance of reinforcement learning algorithms. It acquires comprehensive performance data by conducting multiple rounds of testing, including training and adversarial tests, on multiple reinforcement learning algorithms in a target adversarial environment. Then, based on a hierarchical evaluation index system covering time response and convergence performance indicators, reward function performance indicators, and adversarial performance indicators, it calculates the adversarial performance of each reinforcement learning algorithm, obtaining a comprehensive adversarial performance score. Finally, it achieves scenario-based algorithm selection based on the adversarial performance score. This device comprehensively characterizes the overall performance of reinforcement learning algorithms in adversarial scenarios from multiple dimensions, including time response, training convergence, reward acquisition, and actual adversarial performance, avoiding the one-sidedness of evaluating based on a single index and improving the reliability of reinforcement learning algorithm selection in adversarial game scenarios.

[0095] In one embodiment, the performance calculation module 520 is specifically used for: Based on the hierarchical evaluation index system and the performance data, the adversarial effectiveness of each reinforcement learning algorithm is calculated to determine the adversarial effectiveness score of each algorithm, including: According to the hierarchical evaluation index system, the performance data of each reinforcement learning algorithm are calculated to determine the initial index data of each reinforcement learning algorithm; Based on the architecture type information of each reinforcement learning algorithm, the indicator items that are not applicable to each reinforcement learning algorithm in the hierarchical evaluation indicator system are determined, and the inapplicable indicator items are assigned a preset benchmark value to obtain the adaptive indicator data. The comprehensive weight of each indicator in the hierarchical evaluation index system is determined, and the adaptation index data is weighted based on the comprehensive weight to obtain the adversarial effectiveness score of each reinforcement learning algorithm.

[0096] In one embodiment, the performance calculation module 520 is further configured to: Determining the comprehensive weight of each indicator in the hierarchical evaluation index system includes: Based on the judgment matrix constructed using the preset scaling method, the feature vectors of each indicator are determined, and the feature vectors are normalized to obtain the first weight of each indicator. Based on the performance data of each reinforcement learning algorithm on various indicators, the data information entropy of each indicator is determined, and the second weight of each indicator is determined based on the data information entropy. The first weight and the second weight are linearly weighted and fused to obtain the comprehensive weight of each indicator.

[0097] In one embodiment, the performance calculation module 520 is further configured to: The time response and convergence performance metrics are determined by the average response time of the environment simulation, the variance of the algorithm execution response time, the value function loss, and the maximum policy loss. The performance metrics of the reward function include cumulative reward data, average reward data, and reward fluctuation data. The performance indicators for combat include win rate in multiple rounds of combat, enemy destruction rate, and our own loss rate.

[0098] In one embodiment, the performance calculation module 520 is further configured to: The multi-round adversarial win rate is determined based on the number of wins and losses of each reinforcement learning algorithm in the multi-round adversarial test; The enemy destruction rate is determined based on the ratio of the number of enemies destroyed in each round of multi-round combat testing to the initial number of enemies; The stated loss rate is determined based on the ratio of the number of losses suffered by our side in each round of multi-round adversarial testing to our initial number.

[0099] In one embodiment, the performance calculation module 520 is further configured to: The cumulative reward data is determined by statistically analyzing the immediate rewards of each reinforcement learning algorithm according to the test time step; The average reward data is determined by dividing the cumulative reward by the total number of time steps in a single test cycle; The reward fluctuation data is determined based on the variance of each instant reward relative to the average reward data within a single test cycle.

[0100] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute an algorithm selection method based on the adversarial performance of reinforcement learning algorithms. This method includes: performing multiple rounds of testing on multiple reinforcement learning algorithms in a target adversarial environment, and obtaining performance data of each reinforcement learning algorithm during the testing process; the testing includes training testing and adversarial testing. Based on the hierarchical evaluation index system and the performance data, the adversarial effectiveness of each reinforcement learning algorithm is calculated to determine the adversarial effectiveness score of each reinforcement learning algorithm. The hierarchical evaluation index system is constructed based on the environmental simulation parameters of the target adversarial environment and the algorithm training parameters of each reinforcement learning algorithm. The hierarchical evaluation index system includes time response and convergence performance index, reward function performance index, and adversarial performance index. The time response and convergence performance index is used to characterize the response time characteristics of the algorithm's interaction with the environment and the convergence characteristics of the loss function during training. The reward function performance index is used to characterize the reward accumulation characteristics and reward fluctuation characteristics of the algorithm during adversarial process. The adversarial performance index is used to characterize the adversarial combat capability of the algorithm. Based on the adversarial effectiveness score, an algorithm selection result suitable for the target adversarial environment is determined from the plurality of reinforcement learning algorithms.

[0101] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0102] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the algorithm selection method based on the adversarial effectiveness of reinforcement learning algorithms provided by the above methods. The method includes: performing multiple rounds of testing on multiple reinforcement learning algorithms in a target adversarial environment to obtain performance data of each reinforcement learning algorithm during the testing process; the testing includes training testing and adversarial testing. Based on the hierarchical evaluation index system and the performance data, the adversarial effectiveness of each reinforcement learning algorithm is calculated to determine the adversarial effectiveness score of each reinforcement learning algorithm. The hierarchical evaluation index system is constructed based on the environmental simulation parameters of the target adversarial environment and the algorithm training parameters of each reinforcement learning algorithm. The hierarchical evaluation index system includes time response and convergence performance index, reward function performance index, and adversarial performance index. The time response and convergence performance index is used to characterize the response time characteristics of the algorithm's interaction with the environment and the convergence characteristics of the loss function during training. The reward function performance index is used to characterize the reward accumulation characteristics and reward fluctuation characteristics of the algorithm during adversarial process. The adversarial performance index is used to characterize the adversarial combat capability of the algorithm. Based on the adversarial effectiveness score, an algorithm selection result suitable for the target adversarial environment is determined from the plurality of reinforcement learning algorithms.

[0103] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program is implemented to perform the algorithm selection method based on the adversarial effectiveness of reinforcement learning algorithms provided by the above methods. The method includes: performing multiple rounds of testing on multiple reinforcement learning algorithms in a target adversarial environment, and obtaining performance data of each reinforcement learning algorithm during the testing process; the testing includes training testing and adversarial testing. Based on the hierarchical evaluation index system and the performance data, the adversarial effectiveness of each reinforcement learning algorithm is calculated to determine the adversarial effectiveness score of each reinforcement learning algorithm. The hierarchical evaluation index system is constructed based on the environmental simulation parameters of the target adversarial environment and the algorithm training parameters of each reinforcement learning algorithm. The hierarchical evaluation index system includes time response and convergence performance index, reward function performance index, and adversarial performance index. The time response and convergence performance index is used to characterize the response time characteristics of the algorithm's interaction with the environment and the convergence characteristics of the loss function during training. The reward function performance index is used to characterize the reward accumulation characteristics and reward fluctuation characteristics of the algorithm during adversarial process. The adversarial performance index is used to characterize the adversarial combat capability of the algorithm. Based on the adversarial effectiveness score, an algorithm selection result suitable for the target adversarial environment is determined from the plurality of reinforcement learning algorithms.

[0104] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0105] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An algorithm selection method based on the adversarial effectiveness of reinforcement learning algorithms, characterized in that, include: Multiple reinforcement learning algorithms were tested in a target adversarial environment to obtain performance data of each algorithm during the testing process. The tests include training tests and adversarial tests; Based on the hierarchical evaluation index system and the performance data, the adversarial effectiveness of each reinforcement learning algorithm is calculated to determine the adversarial effectiveness score of each reinforcement learning algorithm. The hierarchical evaluation index system is constructed based on the environmental simulation parameters of the target adversarial environment and the algorithm training parameters of each reinforcement learning algorithm. The hierarchical evaluation index system includes time response and convergence performance index, reward function performance index, and adversarial performance index. The time response and convergence performance index is used to characterize the response time characteristics of the algorithm's interaction with the environment and the convergence characteristics of the loss function during training. The reward function performance index is used to characterize the reward accumulation characteristics and reward fluctuation characteristics of the algorithm during adversarial process. The adversarial performance index is used to characterize the adversarial combat capability of the algorithm. Based on the adversarial effectiveness score, an algorithm selection result suitable for the target adversarial environment is determined from the plurality of reinforcement learning algorithms.

2. The algorithm selection method based on the adversarial effectiveness of reinforcement learning algorithms according to claim 1, characterized in that, Based on the hierarchical evaluation index system and the performance data, the adversarial effectiveness of each reinforcement learning algorithm is calculated to determine the adversarial effectiveness score of each algorithm, including: According to the hierarchical evaluation index system, the performance data of each reinforcement learning algorithm are calculated to determine the initial index data of each reinforcement learning algorithm; Based on the architecture type information of each reinforcement learning algorithm, the indicator items that are not applicable to each reinforcement learning algorithm in the hierarchical evaluation indicator system are determined, and the inapplicable indicator items are assigned a preset benchmark value to obtain the adaptive indicator data. The comprehensive weight of each indicator in the hierarchical evaluation index system is determined, and the adaptation index data is weighted based on the comprehensive weight to obtain the adversarial effectiveness score of each reinforcement learning algorithm.

3. The algorithm selection method based on the adversarial effectiveness of reinforcement learning algorithms according to claim 2, characterized in that, Determining the comprehensive weight of each indicator in the hierarchical evaluation index system includes: Based on the judgment matrix constructed using the preset scaling method, the feature vectors of each indicator are determined, and the feature vectors are normalized to obtain the first weight of each indicator. Based on the performance data of each reinforcement learning algorithm on various indicators, the data information entropy of each indicator is determined, and the second weight of each indicator is determined based on the data information entropy. The first weight and the second weight are linearly weighted and fused to obtain the comprehensive weight of each indicator.

4. The algorithm selection method based on the adversarial effectiveness of reinforcement learning algorithms according to claim 1, characterized in that, The time response and convergence performance metrics include the average response time of the environmental simulation, the variance of the algorithm execution response time, the value function loss, and the maximum policy loss. The performance metrics of the reward function include cumulative reward data, average reward data, and reward fluctuation data. The performance indicators for combat include win rate in multiple rounds of combat, enemy destruction rate, and our own loss rate.

5. The algorithm selection method based on the adversarial effectiveness of reinforcement learning algorithms according to claim 4, characterized in that, The multi-round adversarial win rate is determined based on the number of wins and losses of each reinforcement learning algorithm in the multi-round adversarial test; The enemy destruction rate is determined based on the ratio of the number of enemies destroyed in each round of multi-round combat testing to the initial number of enemies; The stated loss rate is determined based on the ratio of the number of losses suffered by our side in each round of multi-round adversarial testing to our initial number.

6. The algorithm selection method based on the adversarial effectiveness of reinforcement learning algorithms according to claim 4, characterized in that, The cumulative reward data is determined by statistically analyzing the immediate rewards of each reinforcement learning algorithm according to the test time step; The average reward data is determined by dividing the cumulative reward by the total number of time steps in a single test cycle; The reward fluctuation data is determined based on the variance of each instant reward relative to the average reward data within a single test cycle.

7. An algorithm selection device based on the adversarial effectiveness of reinforcement learning algorithms, characterized in that, include: The testing module is used to perform multiple rounds of testing on various reinforcement learning algorithms in a target adversarial environment, and to obtain performance data of each reinforcement learning algorithm during the testing process; the testing includes training testing and adversarial testing. The performance calculation module is used to calculate the adversarial performance of each reinforcement learning algorithm based on the hierarchical evaluation index system and the performance data, and to determine the adversarial performance score of each reinforcement learning algorithm. The hierarchical evaluation index system is constructed based on the environmental simulation parameters of the target adversarial environment and the algorithm training parameters of each reinforcement learning algorithm. The hierarchical evaluation index system includes time response and convergence performance index, reward function performance index, and adversarial performance index. The time response and convergence performance index is used to characterize the response time characteristics of the algorithm's interaction with the environment and the convergence characteristics of the loss function during training. The reward function performance index is used to characterize the reward accumulation characteristics and reward fluctuation characteristics of the algorithm during adversarial process. The adversarial performance index is used to characterize the adversarial combat capability of the algorithm. The selection module is used to determine the algorithm selection result that is suitable for the target adversarial environment from the plurality of reinforcement learning algorithms based on the adversarial effectiveness score.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the algorithm selection method based on the adversarial effectiveness of reinforcement learning algorithms as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the algorithm selection method based on the adversarial effectiveness of reinforcement learning algorithms as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the algorithm selection method based on the adversarial effectiveness of reinforcement learning algorithms as described in any one of claims 1 to 6.

Images