Combat intelligent agent reinforcement learning training method and system based on self-game mode

Abstract

The application discloses a combat intelligent agent reinforcement learning training method and system based on a self-game mode and belongs to the technical field of twin modeling, and comprises the following steps: based on a combat target, dynamically obtaining priority weights of various tactical indexes, based on the priority weights, comprehensively evaluating various tactical indexes of a historical combat strategy, and screening out a current optimal strategy; in a self-game process, based on various tactical indexes of an intelligent agent and corresponding priority weights, a comprehensive strategy evaluation result of each combat is generated; when the number of self-game combats reaches a preset first number threshold, based on the comprehensive strategy evaluation result within the first number threshold, the effectiveness of a current combat strategy is dynamically determined, if the current combat strategy is determined to be effective, the current combat strategy is replaced with the optimal strategy, and if the current combat strategy is determined to be ineffective, based on various tactical indexes and the comprehensive strategy evaluation result within the first number threshold, a combat strategy dynamic adjustment is performed.

Description

Technical Field

[0001] This invention relates to the field of twin modeling technology, and in particular to a method and system for training combat intelligent agents based on self-game mode reinforcement learning. Background Technology

[0002] The essence of combat intelligent agent adversarial training lies in constructing a data-driven, continuously evolving "virtual war laboratory." Its core method utilizes a multi-agent reinforcement learning framework to organize intelligent agents in high-intensity autonomous adversarial games within a high-fidelity, variable simulated battlefield environment built on digital twin and generative artificial intelligence technologies. This system automatically generates large-scale interactive experience data, which, combined with historical knowledge and equipment information, is filtered and optimized through algorithms such as course learning and adversarial sampling, thus forming the training foundation for continuous strategy iteration. The ultimate goal of this closed-loop training system is to efficiently incubate intelligent combat clusters capable of adapting to complex and uncertain real battlefield environments, possessing superior autonomous coordination, adaptability, and adversarial capabilities.

[0003] The rise of combat intelligent agent adversarial training is a key path for the development of military intelligence. Its technological foundation relies primarily on three pillars: First, a theoretical framework centered on multi-agent reinforcement learning, enabling agents to autonomously optimize their strategies through trial and error with the environment and adversaries. Second, high-fidelity, scalable virtual battlefield environment construction technologies, represented by digital twins and generative artificial intelligence, provide a simulation platform that closely approximates reality and can automatically generate massive amounts of tactical scenarios. Third, system-level support comprised of distributed simulation and large-scale parallel computing ensures the efficient operation of complex adversarial simulations. Furthermore, advanced machine learning methods such as curriculum learning and adversarial sampling are used to guide the training process, enabling agents to evolve progressively from easy to difficult, from simple adversarial to complex collaborative operations, ultimately gaining strong adaptability and collaborative combat capabilities in a dynamic and uncertain battlefield environment.

[0004] For example, the invention patent announcement CN118228781B discloses a training system based on generative adversarial networks, which includes: a training setting module, a data generation module, a data judgment module, and a training control module. The training setting module is used to set the training content, the data generation module generates data based on the set training content, the data judgment module is used to judge the generated data, and the training control module is used to control and manage the training process.

[0005] For example, the invention patent announcement CN118662913B discloses an AI-based large-scale model-based auxiliary decision-making method and system, which includes: collecting training data and encoding it into observation vectors; an agent selecting a first action based on the information collected; the agent receiving a reward after executing the action and storing the experience in an experience replay pool; the agent obtaining the training start time and the number of collected experiences through a trained first neural network; and collecting experience to start training after reaching the start time.

[0006] However, in the process of implementing the inventive technical solution in the embodiments of this application, it was found that the above-mentioned technology has at least the following technical problems: In existing technologies, the learning parameters for training combat agents cannot adapt to changes in the training process, resulting in insufficient adaptability of reinforcement learning training, low efficiency in strategy optimization, inability to dynamically respond to different combat objectives and needs, slow convergence of self-game training, or unstable results. Summary of the Invention

[0007] To address the problems in existing technologies where the learning parameters for training combat agents cannot adapt to changes in the training process, resulting in insufficient adaptability and low efficiency in strategy optimization, an inability to dynamically respond to different combat objectives and requirements, and slow convergence or unstable results in self-game training, this invention provides a method and system for training combat agents based on a self-game model. The technical solution is as follows: On the one hand, a reinforcement learning training method for combat agents based on a self-game model is provided. This method includes: dynamically acquiring the priority weights of various tactical indicators based on the combat objective, including but not limited to reward acquisition, casualty rate, troop loss ratio, and position retention rate; comprehensively evaluating various tactical indicators of historical combat strategies based on the priority weights, and selecting the current optimal strategy based on the comprehensive evaluation results; engaging in self-game combat with the current combat strategy and the current optimal strategy, and generating a comprehensive strategy evaluation result for each combat operation in real time based on the agent's various tactical indicators and corresponding priority weights during the self-game process; when the number of self-game combat operations reaches a preset first threshold, dynamically determining the effectiveness of the current combat strategy based on the comprehensive strategy evaluation results within the first threshold; if the current combat strategy is determined to be effective, replacing it with the optimal strategy; if the current combat strategy is determined to be invalid, dynamically adjusting the combat strategy based on the various tactical indicators and the comprehensive strategy evaluation results within the first threshold.

[0008] On the other hand, a reinforcement learning training system for combat agents based on a self-game model is provided. This system is applied to a reinforcement learning training method for combat agents based on a self-game model. The system includes: an optimal strategy selection module, a combat strategy evaluation module, and a combat strategy training module. The optimal strategy selection module dynamically acquires the priority weights of various tactical indicators based on the combat objective, comprehensively evaluates the various tactical indicators of historical combat strategies based on the priority weights, and selects the current optimal strategy based on the comprehensive evaluation results. The combat strategy evaluation module engages in self-game combat using the current combat strategy and the current optimal strategy, and generates a comprehensive strategy evaluation result for each combat operation in real time based on the agent's various tactical indicators and corresponding priority weights during the self-game process. The combat strategy training module dynamically determines the effectiveness of the current combat strategy based on the comprehensive strategy evaluation results within the first threshold number of self-game operations. If the current combat strategy is determined to be effective, it is replaced with the optimal strategy; if the current combat strategy is determined to be invalid, the combat strategy is dynamically adjusted based on the various tactical indicators and the comprehensive strategy evaluation results within the first threshold number.

[0009] Beneficial effects The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: 1. This invention dynamically acquires the priority weights of tactical indicators and accurately selects the optimal strategy. It conducts self-game training and dynamically judges the effectiveness of the strategy according to a threshold. It simultaneously adjusts key parameters such as entropy regularization coefficient, discount factor, and training number threshold, thereby improving the pertinence and accuracy of combat strategy evaluation. It optimizes the efficiency, convergence, and adaptability of strategy learning in the reinforcement learning process, and iterates effective strategies in a timely manner while avoiding ineffective strategies. In this way, it achieves a deep fit between the reinforcement learning training of combat agents and actual combat objectives, requirements, and stages, creating more efficient combat strategies with greater practical value.

[0010] 2. This invention combines basic weights for matching combat objectives, obtains dynamic adjustment quantities based on quantitative parameters of combat requirements, and determines correction coefficients according to combat stages. Through weighted combination and normalization processing, it generates priority weights for tactical indicators, thereby achieving precise adaptation of weights to combat objectives, real-time requirements, and stage priorities. This avoids the limitations of fixed weights and realizes the dynamic and refined construction of tactical indicator weights, providing a scientific and practical core basis for subsequent comprehensive evaluation of combat strategies.

[0011] 3. This invention selects historical strategies that meet the combat objectives as initial targets, standardizes the actual values ​​of the indicators by combining them with preset tactical indicator evaluation thresholds, and then uses priority weights to weighted summation of the standard values ​​to obtain comprehensive evaluation indicators, which are then sorted and selected. This process eliminates interference from irrelevant strategies, ensuring that the evaluation results are aligned with the current objectives and are quantifiable and controllable. This improves the objectivity and relevance of strategy evaluation, thereby enabling the accurate extraction of the optimal strategy adapted to the current scenario from historical strategies and providing a high-quality benchmark for subsequent self-game training.

[0012] 4. This invention employs a three-step progressive judgment logic: win rate, average achievement rate of key tactical indicators, and decay rate of comprehensive evaluation indicators. It performs layered verification of the self-game results within the first threshold number, first filtering out low-win-rate strategies, then verifying the achievement of core indicators, and finally investigating the decay of strategy effects. This avoids the one-sidedness of single-dimensional judgment, reduces the risk of misjudging ineffective strategies, and improves the comprehensiveness and accuracy of strategy effectiveness judgment. In turn, it achieves a scientific identification of the practical adaptability of current combat strategies, provides a reliable decision-making basis for subsequent strategy replacement or dynamic adjustment, and ensures the efficient advancement of reinforcement learning training.

[0013] 5. This invention dynamically adjusts the discount factor, entropy regularization coefficient, and first number threshold in multiple dimensions based on the results of self-game analysis. Each adjustment process relies on a preset mapping table to achieve quantitative adaptation, thereby enabling the core hyperparameters and training cycle to dynamically adapt to the training state. This balances strategy exploration and convergence, as well as immediate and long-term benefits, avoiding the inefficiency caused by parameter fixation. As a result, it achieves adaptive optimization of the reinforcement learning training process of combat intelligent agents, significantly improving training efficiency and strategy adaptability, and providing support for the efficient generation of high-quality combat strategies.

[0014] 6. By setting a second threshold, combat strategies that have reached the threshold in the cumulative self-game combat but are still determined to be invalid are marked as invalid strategies. This allows for the timely termination of the subsequent training process of invalid strategies, avoiding the meaningless consumption of training resources such as computing power and time, and eliminating the interference of invalid strategies on the training process. This improves the resource utilization efficiency of combat intelligent agent reinforcement learning training, directs training resources toward the iteration of effective strategies, and ensures the efficient progress of the overall training. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1A flowchart of a reinforcement learning training method for combat intelligent agents based on a self-game mode, provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a combat intelligent agent reinforcement learning training system based on a self-game mode, provided in an embodiment of this application. Detailed Implementation

[0017] The following provides explanations for some of the terms used in this application. It should be noted that these explanations are for the convenience of those skilled in the art and do not constitute a limitation on the scope of protection claimed in this application.

[0018] The embodiments of this application involve at least one, including one or more; where "multiple" means two or more. Furthermore, it should be understood that in the description of this specification, terms such as "first," "second," and "third" are used only for descriptive purposes and should not be construed as indicating relative importance or order. For example, "first device" and "second device" do not represent the degree of importance of the two or their order, but are merely for descriptive distinction. In the embodiments of this application, "and / or" merely describes an association relationship, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0019] The directional terms mentioned in the embodiments of this application, such as "up", "down", "left", "right", "inner", and "outer", are only for reference to the directions in the accompanying drawings. Therefore, the directional terms used are for better and clearer explanation and understanding of the embodiments of this application, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the embodiments of this application.

[0020] References to "one embodiment," "in some examples," or "some embodiments" as described in the embodiments of this application mean that one or more embodiments of this specification include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in some examples," "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0021] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0022] like Figure 1 The diagram shows a flowchart of a reinforcement learning training method for a combat agent based on a self-game model, provided in this application. The method includes the following steps: dynamically acquiring the priority weights of various tactical indicators based on the combat objective, including but not limited to rewards, casualty rate, troop loss ratio, and position retention rate; comprehensively evaluating the various tactical indicators of historical combat strategies based on the priority weights, and selecting the current optimal strategy that matches the combat objective type based on the comprehensive evaluation results; engaging in self-game combat using the current combat strategy and the current optimal strategy, and generating a comprehensive strategy evaluation result for each combat operation in real time based on the agent's various tactical indicators and corresponding priority weights during the self-game process; and when the self-game occurs... When the number of game operations reaches a preset first threshold, the effectiveness of the current operational strategy is dynamically determined based on the comprehensive strategy evaluation results within the first threshold. If the current operational strategy is determined to be effective, it is replaced with the optimal strategy. If the current operational strategy is determined to be ineffective, the operational strategy is dynamically adjusted based on various tactical indicators and comprehensive strategy evaluation results within the first threshold. The dynamic adjustment of the operational strategy means dynamically adjusting the entropy regularization coefficient and discount factor of the operational strategy to improve the learning efficiency and convergence of the operational strategy in subsequent self-game training. The entropy regularization coefficient is a hyperparameter used to adjust the balance between strategy exploration and utilization, and the discount factor is a hyperparameter used to weigh the importance of immediate rewards and future rewards.

[0023] In this embodiment, the present invention first dynamically assigns differentiated priority weights to various tactical indicators such as reward acquisition, casualty rate, troop loss ratio, and position retention rate based on the combat objective. This ensures that the evaluation criteria of tactical indicators are highly aligned with the combat objective, effectively avoiding the problem of evaluation being out of touch with actual combat needs under fixed weights, and guaranteeing the pertinence and rationality of strategy evaluation. Then, based on these priority weights, a comprehensive evaluation of various tactical indicators of historical combat strategies is conducted, and the current optimal strategy that matches the combat objective type is selected. This establishes a high-quality reference benchmark that aligns with combat needs for self-game training, providing a clear direction for subsequent self-game training and significantly improving the effectiveness and adaptability of strategy selection. Subsequently, a self-game is conducted between the current combat strategy and the optimal strategy. During the game, real-time tactical indicators and corresponding priority weights are combined to generate a comprehensive strategy evaluation result for each battle, realizing dynamic monitoring of strategy performance during the self-game process. Precise quantitative monitoring enables timely capture of the tactical performance and overall effectiveness of strategies in actual combat. Finally, when the number of self-play battles reaches a preset threshold, the effectiveness of the current strategy is dynamically determined based on the comprehensive strategy evaluation results within the threshold, effectively distinguishing the actual application value of strategies. Strategies deemed effective are directly replaced with the optimal strategy, achieving rapid accumulation and iteration of high-quality strategies. For strategies deemed ineffective, the entropy regularization coefficient and discount factor are dynamically adjusted based on various tactical indicators and comprehensive evaluation results within the threshold. The adjustment of the entropy regularization coefficient precisely controls the balance between strategy exploration and utilization, while the adjustment of the discount factor reasonably weighs the importance of immediate and future rewards. The synergistic dynamic adjustment of these two factors can specifically address the problems of low efficiency and poor convergence in the learning process of ineffective strategies, significantly improving the learning efficiency and convergence speed of the strategy in subsequent self-play training. Overall, this method integrates the combat objective into the entire process of weight allocation, strategy selection, self-game monitoring, strategy effectiveness determination, and parameter adjustment. Through dynamic evaluation, judgment, and parameter optimization under the self-game model, the reinforcement learning training of the combat agent always revolves around the combat objective. This ensures a high degree of matching between the training process and actual combat needs, and continuously optimizes the performance of the combat strategy through dynamic strategy iteration and parameter adjustment. It effectively improves the relevance, efficiency, and convergence of the reinforcement learning training of the combat agent, enabling the trained combat agent to adapt to different types of combat objectives and possess better combat confrontation capabilities and strategy optimization capabilities.

[0024] Furthermore, based on the current operational target type, the basic priority weights corresponding to each tactical indicator are matched from a pre-set indicator weight mapping table. This table describes the correspondence between different operational target types and the initial weights of each tactical indicator. Based on current operational requirements, operational requirement parameters are dynamically quantified. These parameters include, but are not limited to, mission urgency, troop redundancy, equipment attrition threshold, and tactical risk tolerance. Based on these operational requirement parameters, the adjustment components of each tactical indicator corresponding to each operational requirement parameter are matched from a pre-set indicator weight adjustment mapping table. All adjustment components of each tactical indicator are then summed to obtain the corresponding priority weights. The dynamic adjustment amount of tactical indicators and the indicator weight adjustment mapping relationship table are used to describe the correspondence between various operational requirement parameters and the adjustment components of various tactical indicators. Based on the current operational phase and the type of tactical indicator, the phase correction coefficient of each tactical indicator is determined. Among them, if a tactical indicator is determined to be a key tactical indicator under the current operational phase, it is assigned a preset strengthening correction coefficient for the current operational phase; if it is determined to be a non-key tactical indicator, it is assigned a preset weakening correction coefficient for the current operational phase. For each tactical indicator, its basic priority weight and the corresponding dynamic adjustment amount are weighted and combined to obtain an intermediate weight value. Then, the intermediate weight value is multiplied by the phase correction coefficient to obtain the preliminary correction weight of the indicator. The preliminary correction weights of all tactical indicators are normalized to generate the priority weights of each tactical indicator used for the current comprehensive evaluation.

[0025] In this embodiment, the present invention first matches the basic priority weights of various tactical indicators from a preset indicator weight mapping table based on the current combat objective type. This allows the initial weight allocation to directly anchor the core requirements of the combat objective type, laying a foundation for precise weight optimization in line with actual combat and effectively avoiding the blindness and subjectivity of the initial weight setting. Then, based on the current combat requirements, combat requirement parameters such as mission urgency and troop redundancy parameters are dynamically quantified. The tactical indicator adjustment components corresponding to each parameter are matched from the preset indicator weight adjustment mapping table and summed to obtain the dynamic adjustment amount. This transforms various concrete combat requirements in actual combat into quantifiable weight adjustment bases, allowing weight allocation to accurately respond to dynamic changes in troop strength, equipment, timeliness, tactical risks, and other factors in actual combat. This overcomes the limitation of fixed weights being unable to adapt to dynamic adjustments in combat requirements, giving weight allocation strong flexibility and adaptability to actual scenarios. Simultaneously, a stage correction coefficient is determined based on the current combat phase and tactical indicator type. The correction coefficient is strengthened for key tactical indicators within the combat phase, while the correction coefficient is weakened for non-key tactical indicators, allowing the weight... The weighting is fully aligned with the phased tactical characteristics of the operation, taking into account the differences in tactical priorities at different operational phases. This avoids the bias in indicator evaluation caused by using uniform weights across phases, ensuring that the core tactical indicators of each operational phase highlight their importance in the weighting allocation, and achieving precise phased adaptation of weights. Subsequently, for each tactical indicator, the basic priority weight and dynamic adjustment amount are weighted together to obtain an intermediate weight value, which is then multiplied by the phase correction coefficient to obtain the preliminary corrected weight. Through a hierarchical weight fusion method, the basic requirements of the operational target type, the dynamic impact of real-time operational requirements, and the tactical priorities of the operational phase are organically combined, allowing the preliminary corrected weight to comprehensively reflect the weighting requirements of multi-dimensional combat factors, effectively avoiding the one-sidedness caused by a single factor dominating the weighting allocation. Finally, the preliminary corrected weights of all tactical indicators are normalized, ensuring that the weights of each tactical indicator are on a uniform quantitative scale, guaranteeing the standardization, comparability, and scientific nature of the weights, and avoiding the distortion of subsequent comprehensive evaluation results due to differences in the dimensions and values ​​of the preliminary corrected weights of each indicator. This ensures that the final generated priority weights have practical evaluation and application value.Overall, this method constructs a refined and dynamic weighting system that integrates basic weights to anchor targets, dynamically adjusts to adapt to needs, adjusts to key points in stages, integrates multi-dimensional weighting, and normalizes and standardizes quantification. It transforms multi-dimensional core factors of actual combat, such as combat target type, real-time combat needs, and characteristics of combat stages, into quantifiable and adjustable weight influencing factors. This enables the customized generation of tactical indicator priority weights, completely eliminating the drawbacks of traditional fixed weights or single-factor weighting. The resulting priority weights can comprehensively and accurately match the current combat target positioning, real-time needs, and stage characteristics. From the source, it provides a scientific, accurate, and highly practical weight basis for the comprehensive evaluation of subsequent combat strategies, the selection of optimal strategies, and the dynamic evaluation in self-game training. This effectively ensures the pertinence, accuracy, and effectiveness of the strategy evaluation stage in the entire self-game training process.

[0026] Furthermore, based on priority weights, the steps of comprehensively evaluating various tactical indicators of historical combat strategies and selecting the current optimal strategy based on the comprehensive evaluation results include: selecting combat strategies that conform to the current combat objective type from historical combat strategies as preliminary combat strategies; obtaining preset tactical indicator evaluation thresholds, including but not limited to reward thresholds, casualty rate thresholds, troop loss ratio thresholds, and position retention rate thresholds; for each preliminary combat strategy, comparing the actual values ​​of its various tactical indicators with the corresponding tactical indicator evaluation thresholds to obtain the standard values ​​of each tactical indicator, and then using priority weights to perform a weighted summation of the standard values ​​of each tactical indicator to obtain the comprehensive strategy evaluation index of the preliminary combat strategy; ranking all preliminary combat strategies from largest to smallest according to their comprehensive strategy evaluation index, and determining the preliminary combat strategy ranked first as the current optimal strategy.

[0027] In this embodiment, the present invention provides a method for comprehensively evaluating historical combat strategies and selecting the optimal current strategy based on priority weights. First, it selects combat strategies that match the type of combat objective from historical strategies as preliminary combat strategies. By filtering out invalid strategies that do not match the current combat objective in advance, the computational scope and workload of subsequent comprehensive evaluation are significantly reduced, effectively avoiding meaningless evaluation operations. This ensures that the scope of strategy evaluation is precisely anchored to the core needs of the current combat objective, laying the foundation for efficient and accurate strategy evaluation in the future. Next, it obtains preset threshold values ​​for tactical indicators such as reward acquisition and casualty rate, establishing a unified and quantifiable objective evaluation benchmark for tactical indicators with different attributes and dimensions. This solves the problem that various tactical indicators cannot be directly compared and calculated due to differences in their numerical characteristics and evaluation dimensions, ensuring the scientific and rational nature of subsequent indicator standardization processing. This provides a unified, quantifiable basis for evaluating the actual performance of each tactical indicator. Finally, for each preliminary combat strategy, it performs a ratio calculation between the actual values ​​of its various tactical indicators and the corresponding evaluation threshold to obtain a standard value. This ratio calculation achieves the normalization of the actual values ​​of each tactical indicator, completely eliminating the dimensional differences between different indicators. This makes previously incompatible tactical indicators additive. Combined with the dynamically generated priority weights tailored to the current full-dimensional operational needs, the standard values ​​of various tactical indicators are weighted and summed to obtain a comprehensive strategy evaluation index. This ensures that the comprehensive evaluation index is not simply a numerical summation of various tactical indicators, but a quantitative result that accurately reflects the differences in importance of each indicator under the current operational objectives, real-time needs, and stage characteristics. This allows the comprehensive evaluation results to truly and objectively reflect the comprehensive operational effectiveness of each initial operational strategy in the current combat scenario, ensuring the relevance, accuracy, and scientific nature of the strategy evaluation. Finally... All preliminary operational strategies were ranked from highest to lowest according to comprehensive strategy evaluation indicators, and the strategy ranked first was determined as the current optimal strategy. This quantitative ranking method eliminated the drawbacks of subjective judgment in traditional strategy selection, making the selection process of the optimal strategy highly objective and unique. It can accurately identify the strategy with the best comprehensive operational effectiveness from the preliminary strategies that are in line with the current operational objectives, and establish a high-quality, highly relevant benchmark for subsequent self-play combat training. This provides a clear direction for optimization in subsequent self-play adversarial training and effectively avoids the problem of the training process deviating from the current operational objectives. Overall, this method constructs a standardized, scientific, and fully quantitative historical combat strategy evaluation and screening system, which includes target selection, threshold calibration, quantitative transformation, weighted synthesis, and ranking and optimization. The entire process replaces subjective judgment with objective quantitative calculations, and deeply integrates the precise priority weights dynamically generated above into the entire evaluation process. This not only significantly improves the efficiency and accuracy of historical strategy evaluation and screening, but also provides high-quality and suitable optimal reference strategies for subsequent self-game training, fundamentally ensuring the effectiveness of subsequent self-game adversarial training, and further enhancing the pertinence and efficiency of combat intelligent agent reinforcement learning training.

[0028] Furthermore, the steps for dynamically determining the effectiveness of the current combat strategy based on the comprehensive strategy evaluation results within the first threshold number include: Step 1: Statistically calculate the winning percentage of the current combat strategy in the self-game combat that reaches the first threshold number, and determine whether the winning percentage exceeds the preset winning percentage threshold. If not, the current combat strategy is directly determined to be invalid; if so, proceed to Step 2. Step 2: Calculate the average achievement rate of the preset key tactical indicators in the self-game combat, and determine whether the average achievement rate of the key tactical indicators reaches the preset indicator achievement rate threshold. If not, the current combat strategy is determined to be invalid; if so, proceed to Step 3. Step 3: Based on the comprehensive strategy evaluation indicators of each combat in the self-game combat, calculate their decay rate with the combat sequence, and determine whether the decay rate of the comprehensive evaluation indicators exceeds the preset indicator decay rate threshold. If so, the current combat strategy is determined to be invalid; otherwise, the current combat strategy is determined to be effective.

[0029] In this embodiment, the present invention constructs a multi-dimensional, progressive combat strategy effectiveness judgment system, coupled with a targeted hyperparameter and threshold dynamic adjustment mechanism, to achieve a closed-loop linkage between accurate strategy effectiveness judgment and scientific optimization of training parameters in self-game training. This significantly improves the scientific nature of the judgment results and the accuracy of the adjustment operations, ensuring the efficiency and quality of reinforcement learning training for combat agents from the core aspects. Specifically, the dynamic judgment step based on the comprehensive strategy evaluation results within the first threshold uses a three-step progressive judgment logic to achieve a comprehensive and in-depth effectiveness detection of the current combat strategy: The first step involves statistically analyzing the win rate of self-game operations and comparing it with a preset threshold, completing a preliminary screening from the overall adversarial results level. This quickly eliminates strategies with unsatisfactory overall win rates, reducing the computational load of subsequent fine-grained judgments and significantly improving judgment efficiency, while anchoring the core basic indicator of the strategy's overall adversarial capability; the second step, under the premise of a satisfactory win rate, further calculates the average achievement rate of key tactical indicators and verifies whether they meet the standards, effectively avoiding... This approach avoids the one-sidedness of solely relying on win rate as the criterion, neglecting the core tactical requirements of operational objectives. It ensures that strategies that pass the evaluation not only achieve good results in combat but also accurately align with the key tactical needs of the current operation, balancing combat effectiveness with the achievement of tactical goals. The third step, based on achieving both win rate and key tactical indicators, calculates the decay rate of comprehensive strategy evaluation indicators with the sequence of operations and performs threshold verification. This accurately detects the long-term operational stability and sustainability of the strategy's effectiveness, avoiding the selection of short-lived strategies that perform well initially but rapidly decline in effectiveness later. It ensures that strategies ultimately deemed effective possess long-term, stable operational effectiveness. These three progressive steps, from the overall picture to the details, from results to sustainability, form a multi-dimensional, comprehensive effectiveness evaluation system. This completely eliminates the drawbacks of single-dimensional evaluation, making strategy effectiveness evaluation results more accurate, comprehensive, and scientific, providing a reliable and objective basis for subsequent decision-making regarding whether to implement strategy replacement or dynamic adjustment.For the dynamic adjustment phase of strategies deemed invalid, multi-dimensional and targeted adjustments are made based on actual combat training data. This ensures that the optimization of hyperparameters and thresholds is supported by clear data, avoiding blind adjustments. The discount factor is dynamically adjusted based on average reward acquisition and average achievement rate of key tactical indicators. This deeply links the adjustment of the hyperparameter to the actual reward acquisition level and the achievement of core tactical indicators, allowing the optimization of the discount factor to specifically balance the importance of immediate and future rewards. This precisely adapts to the strategy's shortcomings in reward pursuit and achievement of key tactical objectives, improving the strategy's efficiency in pursuing combat rewards and tactical goals. The comprehensive strategy evaluation index dynamically adjusts the entropy regularization coefficient, ensuring that the adjustment of this hyperparameter aligns with the overall combat performance of the strategy. This allows for precise control of the balance between strategy exploration and utilization, specifically addressing issues such as optimization stagnation due to insufficient strategy exploration or low learning efficiency due to overexploration. Furthermore, the first-order threshold is dynamically adjusted based on the decay rate of the comprehensive strategy evaluation index and the strategy's adaptability to the current optimal strategy. This ensures that the threshold setting matches the actual learning pace and adversarial adaptability of the strategy, avoiding premature or delayed effectiveness determination due to fixed thresholds. This allows subsequent strategy effectiveness determinations to better align with the actual training process of the strategy, improving the scenario adaptability of the determination process. Overall, this method deeply integrates multi-dimensional progressive validity assessment with precise dynamic adjustment across multiple dimensions. First, it accurately identifies the validity status of a strategy through a scientific assessment system. Then, it optimizes ineffective strategies using targeted hyperparameters and thresholds. Simultaneously, it directly replaces effective strategies with optimal ones, rapidly accumulating high-quality strategies. This forms a virtuous cycle of rapid iteration of effective strategies and precise optimization of ineffective strategies, continuously driving performance improvement in combat strategies. This collaborative mechanism of assessment and adjustment makes strategy validity assessment in self-game training more scientific, hyperparameter and threshold adjustments more precise, and the training process more adaptable to the actual performance of the strategies. It not only significantly improves the learning efficiency and convergence of reinforcement learning training for combat agents but also ensures that the trained combat strategies possess high win rates, meet core tactical standards, and have stable and sustainable comprehensive combat effectiveness. This further enhances the practical adaptability and strategy optimization capabilities of the self-game training model, ensuring that the training of combat agents always efficiently advances around the needs of actual combat.

[0030] Furthermore, the steps for dynamically adjusting the operational strategy include: dynamically adjusting the discount factor based on the average reward and average achievement rate of key tactical indicators for each operation within the first threshold; dynamically adjusting the entropy regularization coefficient based on the comprehensive strategy evaluation index for each operation within the first threshold; and dynamically adjusting the first threshold based on the decay rate of the comprehensive strategy evaluation index with the order of operations and the adaptability measure of the current operational strategy to the current optimal strategy within the first threshold.

[0031] In this embodiment, the present invention uses actual quantitative data from self-game combat within the first threshold as its core basis. It conducts multi-dimensional and targeted precise dynamic adjustments to three core elements: discount factor, entropy regularization coefficient, and the first threshold. This completely abandons the rigid mode of fixed hyperparameters and decision thresholds in traditional reinforcement learning training, ensuring that all adjustment operations are deeply bound to the actual self-game performance of the combat strategy. This achieves data-driven adaptation and optimization of hyperparameters and decision thresholds, laying a solid scientific foundation of parameters and thresholds for the efficient and precise advancement of subsequent self-game training. Specifically, it dynamically adjusts the average reward and average achievement rate of key tactical indicators for each combat within this threshold. Adjusting the discount factor directly links the adjustment of this hyperparameter to the actual reward acquisition level and the achievement of core tactical objectives. This allows for a targeted balance between the importance of immediate and future rewards, effectively avoiding the problems caused by a fixed discount factor, such as strategies excessively pursuing short-term rewards while neglecting long-term tactical planning, or excessively focusing on future rewards while missing immediate combat advantages. The discount factor value precisely matches the actual needs of the strategy in reward accumulation and the achievement of key tactical objectives, ensuring that the strategy can balance immediate combat effectiveness and long-term tactical planning in subsequent learning, thus improving the strategy's alignment with combat objectives. The entropy positive factor is dynamically adjusted based on the comprehensive strategy evaluation index of each combat operation within this threshold. The entropy regularization coefficient, based on the core quantitative value reflecting the overall comprehensive combat effectiveness of the strategy, can accurately control the balance between strategy exploration and utilization. It can specifically address the learning stagnation and difficulty in discovering better combat strategies caused by insufficient strategy exploration, while avoiding the problems of low strategy learning efficiency and slow convergence caused by over-exploration. This ensures that the value of the entropy regularization coefficient is highly matched with the current comprehensive combat performance of the strategy, guaranteeing that the strategy maintains a reasonable pace of exploration and utilization in subsequent self-games, and continuously improving the learning and optimization efficiency of the strategy. It is based on the decay rate of the comprehensive strategy evaluation index with the order of combat, and the adaptability measure of the current combat strategy to the current optimal strategy within this threshold. Dynamically adjusting the first threshold combines the decay rate, which reflects the continuous stability of the strategy's operational effectiveness, with the adaptability measure, which reflects the degree of adaptation between the strategy and the optimal strategy, as the basis for threshold adjustment. This allows the threshold for determining the number of times the strategy's effectiveness is determined to be accurately matched with the actual learning pace and adaptability of the strategy. It avoids the problems caused by a fixed threshold, such as determining the strategy's effectiveness too early and failing to collect sufficient operational data, resulting in distorted judgment results, or determining it too late, increasing meaningless computation and delaying strategy iteration and optimization. This makes the setting of the first threshold more in line with the actual training process of the strategy, ensuring the accuracy and efficiency of subsequent strategy effectiveness determinations and significantly reducing the number of ineffective self-play operations.Furthermore, the three adjustment operations are not implemented in isolation, but rather form a synergistic optimization effect through mutual cooperation and support. The precise adjustment of the discount factor and entropy regularization coefficient directly optimizes the learning mechanism of the combat strategy itself, enabling the strategy to have a more scientific learning logic in subsequent self-games. Meanwhile, the dynamic optimization of the first threshold improves the mechanism for judging the effectiveness of the strategy, making subsequent strategy effectiveness detection more accurate and efficient. The synergy between the two makes the closed loop of strategy learning, effectiveness judgment, and strategy adjustment in self-game training more scientific and smoother. Overall, a data-driven, multi-dimensional adaptive hyperparameter and threshold optimization system is constructed, realizing the dynamic iterative optimization of hyperparameters and judgment thresholds related to the combat strategy. This ensures that subsequent self-game training is always carried out in a parameter and threshold environment that adapts to the actual performance of the strategy, effectively improving the learning efficiency, convergence, and combat adaptability of the combat agent's reinforcement learning training, and continuously promoting the iterative upgrade of the combat strategy towards better comprehensive combat effectiveness in self-game training.

[0032] Furthermore, the steps for dynamically adjusting the discount factor include: when the average reward is lower than the preset reward target and the average achievement rate of key tactical indicators has not reached the indicator achievement rate threshold, a first adjustment amount of the discount factor is obtained from a preset first adjustment mapping table based on the difference between the preset reward target and the average reward; a second adjustment amount of the discount factor is obtained from a preset second adjustment mapping table based on the difference between the indicator achievement rate threshold and the average achievement rate of key tactical indicators; and the discount factor is dynamically reduced based on the first and second adjustment amounts of the discount factor, making the strategy more focused on improving short-term returns and immediate tactical execution. When the average reward is lower than the preset reward target and the average achievement rate of key tactical indicators has reached the indicator achievement rate threshold, the discount factor is dynamically reduced based on the difference between the preset reward target and the average reward. The difference is matched with the first adjustment amount of the discount factor from the preset first adjustment mapping table of discount factors. The discount factor is dynamically increased based on the first adjustment amount of the discount factor to strengthen the strategy's consideration of long-term returns and overall task planning. When the average reward is not lower than the preset reward target and the average achievement rate of key tactical indicators has not reached the indicator achievement rate threshold, the second adjustment amount of the discount factor is matched with the second adjustment mapping table of the preset second adjustment table of discount factors based on the difference between the indicator achievement rate threshold and the average achievement rate of key tactical indicators. The discount factor is dynamically increased based on the second adjustment amount of the discount factor to guide the strategy to take into account the achievement of key tactical nodes while pursuing long-term returns. When the average reward is not lower than the preset reward target and the average achievement rate of key tactical indicators reaches the indicator achievement rate threshold, no additional processing is performed.

[0033] In this embodiment, based on the actual performance of the average reward and average achievement rate of key tactical indicators in the first self-game combat within the threshold, the present invention divides four precise scenario-based adjustment logics and constructs a fine-grained adjustment system for discount factors that fits actual combat performance and is adjusted on demand. This completely eliminates the drawback of fixed discount factors being unable to adapt to different combat performances of strategies. The adjustment of discount factors is no longer a general numerical adjustment, but a targeted optimization that is highly matched with the actual shortcomings of the strategy in reward acquisition and core tactical achievement. This achieves precise control of the discount factor on the learning orientation of combat strategies, ensuring that the strategy can adjust the balance logic between immediate and future rewards according to actual combat performance in subsequent self-game training, and always carry out learning in a way that fits the current combat objectives. Specifically, when the average reward received is lower than the preset reward target and the average achievement rate of key tactical indicators fails to reach the threshold, combining the reward difference and tactical indicator difference with a dual adjustment amount and dynamically reducing the discount factor can accurately guide the strategy to focus its learning on improving short-term returns and implementing immediate tactics. This quickly compensates for the strategy's shortcomings in reward acquisition and core tactical achievement in current combat, preventing the strategy from neglecting the core basic requirements of current combat due to excessive consideration of long-term benefits, and achieving rapid optimization of the strategy's immediate combat capabilities. When the average reward received is lower than the preset reward target but the average achievement rate of key tactical indicators meets the target, only the reward difference adjustment amount is used to dynamically increase the discount factor. This can guide the strategy to strengthen its consideration of long-term benefits and overall mission planning while ensuring that the core tactical execution of the strategy meets the target, thus promoting the strategy to take into account both immediate tactical requirements. This study explores operational strategies that enhance long-term reward acquisition, addressing the problem of solely pursuing immediate tactical targets while neglecting long-term benefit accumulation. When average reward acquisition meets targets but average achievement rate of key tactical indicators falls short of thresholds, adjustments are made based solely on the difference in tactical indicators, dynamically increasing the discount factor. This guides the strategy to maintain long-term reward pursuit while simultaneously ensuring the achievement of key tactical nodes. It allows the strategy to prioritize and address shortcomings in core tactical execution during long-term operational planning, preventing the strategy from deviating from core tactical requirements due to excessive pursuit of long-term gains. When both indicators meet preset targets, no additional processing is performed. This maintains the stability of the discount factor while ensuring the strategy's current performance aligns with expectations, avoiding meaningless parameter adjustments that disrupt the strategy's normal learning rhythm and ensuring continuous self-game training under the adapted benefit trade-off logic.Overall, this adjustment method, through scenario-based segmentation, targeted adjustment, and a logic design that ensures stability upon reaching the target, provides clear practical data support and a clear learning orientation for each adjustment of the discount factor. This achieves precise optimization of the strategy's benefit consideration logic while flexibly switching learning orientations based on the strategy's actual performance. It quickly addresses various shortcomings in reward acquisition and core tactical achievement, while avoiding interference from ineffective adjustments. This refined adjustment approach ensures that the discount factor remains highly aligned with the strategy's actual combat performance and the core requirements of the current combat objectives. It guarantees that the strategy's benefit-trading logic in subsequent self-game training always conforms to practical needs, effectively improving the strategy's ability to balance immediate tactical execution and long-term mission planning. This drives the strategy to continuously optimize in self-games, developing high-quality combat strategies that balance immediate combat effectiveness with long-term operational layout, and align with core tactical requirements and overall reward objectives. This further enhances the relevance and effectiveness of the combat agent's reinforcement learning training.

[0034] Furthermore, the steps for dynamically adjusting the entropy regularization coefficient include: Step 4: Determine whether the average comprehensive strategy evaluation index has reached the preset average comprehensive strategy evaluation threshold. If yes, proceed to Step 5; otherwise, mark the current state as a state to be adjusted. Step 5: Determine whether the maximum value of the comprehensive strategy evaluation index has reached the preset maximum threshold for comprehensive strategy evaluation. If yes, proceed to Step 6; otherwise, mark the current state as a state to be adjusted. Step 6: Determine whether the minimum value of the comprehensive strategy evaluation index has reached the preset minimum threshold for comprehensive strategy evaluation. If yes, mark the current state as a normal state; otherwise, mark the current state as a state to be adjusted. If the current state is a normal state, no additional processing is performed. If the current state is a state to be adjusted, the entropy regularization coefficient is increased according to the preset incremental step size.

[0035] In this embodiment, the present invention uses the comprehensive strategy evaluation index of the first self-game combat within the threshold as the core quantitative basis. It designs a three-step progressive quantitative judgment logic from average performance to extreme performance, and constructs a multi-dimensional and comprehensive strategy combat effectiveness state judgment system. This allows the adjustment of the entropy regularization coefficient to have clear and objective quantitative data support, completely eliminating the drawbacks of blindly adjusting hyperparameters or adjusting based on experience. It realizes the on-demand, precise, and controllable adjustment of the entropy regularization coefficient, ensuring that the coefficient is always adapted to the actual comprehensive combat effectiveness of the strategy. It accurately controls the balance between exploration and utilization of the strategy in self-game training, laying a solid core hyperparameter foundation for improving the learning efficiency and convergence optimization of subsequent combat strategies. Step four first determines whether the average comprehensive strategy evaluation index has reached the preset threshold, completing the preliminary verification from the level of the overall operational effectiveness of the strategy. This anchors the overall comprehensive performance of the strategy in self-game training, avoiding misjudgment of the overall state of the strategy due to the effectiveness of a single instance, and ensuring the overall objectivity and scientific nature of the state determination. Step five, based on the average index meeting the standard, further determines whether the maximum value of the comprehensive strategy evaluation index has reached the preset maximum threshold, conducting a second verification from the upper limit of the optimal operational effectiveness of the strategy. This accurately identifies whether the strategy has problems such as acceptable overall average performance but insufficient optimal effectiveness, making it difficult to explore better operational strategies, taking into account both the overall performance and the upper limit potential of the strategy. Step six, assuming that both the average and maximum indicators meet the standard, then determines whether the minimum value of the comprehensive strategy evaluation index has reached the preset minimum threshold, completing the final verification from the lower limit of the worst operational effectiveness of the strategy. This effectively avoids extreme cases where the overall and optimal performance of the strategy meets the standard but there are excessively low effectiveness, and problems such as excessive fluctuations in operational performance, taking into account the stability of the strategy's effectiveness. These three steps of judgment are progressive, from the overall to the extreme value, and from the average performance to the upper and lower limits. They comprehensively cover the core evaluation dimensions of the overall operational effectiveness of the strategy, making the judgment results of the normal state and the state to be adjusted of the strategy more comprehensive, accurate and realistic, and providing a reliable and objective decision-making basis for the subsequent adjustment of the entropy regularization coefficient.Meanwhile, this invention designs differentiated processing logic for different judgment results: if the current state is marked as normal after three-step judgment, it means that the overall combat effectiveness of the strategy has met expectations in the dimensions of overall average, optimal upper limit, and worst lower limit. At this time, no additional processing is performed to maintain the stability of the entropy regularization coefficient, avoid meaningless parameter adjustments that would disrupt the current exploration and utilization balance of the strategy, and ensure that the strategy continues to carry out self-game training in the already achieved effectiveness state, steadily accumulating combat experience and optimizing combat strategies; if it is marked as a state to be adjusted, it means that the overall combat effectiveness of the strategy in at least one dimension has not met the preset requirements, and there is a possibility of adjustment. When encountering problems such as learning bottlenecks and insufficient exploration, the entropy regularization coefficient is increased according to a preset incremental step size. By quantitatively increasing this coefficient, the exploration level of the strategy in self-game training is precisely improved, guiding the strategy to break through the current operational effectiveness bottleneck and actively explore more diverse and better operational strategies. This effectively solves problems such as learning stagnation, difficulty in improving effectiveness, and slow convergence caused by insufficient exploration. Moreover, the preset incremental step size design keeps the improvement of the strategy's exploration level within a controllable range, avoiding the problems of excessive strategy exploration, chaotic operational performance, and low learning efficiency caused by excessively increasing the coefficient. This achieves a precise and moderate improvement in the strategy's exploration level. Overall, this adjustment method employs a refined logical design—a three-step, multi-dimensional judgment process to determine stability upon reaching the target and quantitative adjustment for failure—to deeply bind the adjustment of the entropy regularization coefficient to the actual comprehensive combat effectiveness of the strategy. This achieves a hyperparameter optimization mode where stability is achieved when the strategy is adapted, and exploration is precisely enhanced when bottlenecks are encountered. This ensures that the balance between exploration and utilization adapted to the current combat effectiveness is not arbitrarily disrupted, while also guiding the strategy to increase its exploration level and discover better combat strategies when effectiveness bottlenecks occur. This refined adjustment method ensures that the entropy regularization coefficient is always highly adapted to the actual comprehensive combat effectiveness of the strategy, guaranteeing that the strategy maintains a reasonable and efficient pace of exploration and utilization during self-game training. It effectively avoids the two major problems of learning stagnation due to insufficient exploration and inefficiency due to excessive exploration, significantly improving the learning efficiency and convergence of the combat agent's reinforcement learning training. This drives the strategy to continuously optimize its comprehensive combat effectiveness in self-game, gradually forming a more stable, higher-quality, and more practical combat approach, further strengthening the optimization capability and practical adaptability of the self-game training mode for combat strategies.

[0036] Furthermore, the step of dynamically adjusting the first threshold number includes: when the decay rate of the comprehensive strategy evaluation index with the combat sequence exceeds a preset index decay rate threshold, or the adaptability metric is lower than a preset adaptability metric threshold, then based on the difference between the decay rate and the index decay rate threshold, a first adjustment amount for the threshold number is obtained from a preset threshold number first adjustment mapping table, which defines the correspondence between the decay rate deviation and the recommended training cycle extension amount; based on the difference between the adaptability metric threshold and the adaptability metric, a second adjustment amount for the threshold number is obtained from a preset threshold number second adjustment mapping table, which defines the correspondence between the degree of adaptability deficiency and the recommended training cycle compensation amount; a summation operation is performed based on the first adjustment amount and the second adjustment amount of the discount factor, and the first threshold number is dynamically adjusted and updated based on the summation result; otherwise, no additional processing is performed.

[0037] In this embodiment, the present invention uses the decay rate of the comprehensive strategy evaluation index and the adaptability measurement of the strategy to the optimal strategy as the core judgment and adjustment basis, and constructs a dynamic optimization mechanism for the number of judgments that fits the actual training performance of the strategy. It completely abandons the rigid mode of fixed judgment threshold, and deeply binds the setting of the first judgment threshold with the stability of the strategy's combat effectiveness and the adversarial adaptability. It realizes the dynamic adaptation of the judgment threshold driven by actual combat data, effectively ensuring the accuracy, rationality and efficiency of the subsequent strategy effectiveness judgment link, and laying a solid foundation for the judgment link of the closed-loop optimization of self-game training. The system first uses a binary decision logic—either the attenuation rate exceeding a preset threshold or the adaptability metric falling below a preset threshold—to accurately identify scenarios where the strategy's combat effectiveness rapidly declines with training or its adaptability to the current optimal strategy is insufficient. Threshold adjustment is only initiated in these scenarios requiring supplementary training data, avoiding meaningless parameter adjustments and ensuring the targeted nature of the adjustment. This also ensures that threshold optimization always revolves around solving the core problems in actual strategy training. After adjustment is triggered, the first adjustment amount is matched from a dedicated mapping table based on the difference between the attenuation rate and the preset threshold. This converts the attenuation deviation of the strategy's combat effectiveness into a corresponding extension of the training cycle, specifically compensating for the rapid attenuation caused by the attenuation of effectiveness. To address the issue of insufficient reference value in early-stage training data, this paper extends the decision period to provide the strategy with more opportunities for self-play training, fully collects subsequent combat data, and avoids distortion in strategy effectiveness judgment due to insufficient data. Based on the difference between the adaptability metric and the preset threshold, a second adjustment amount is matched from another dedicated mapping table to the threshold number, transforming the degree of insufficient adversarial adaptation of the strategy into a corresponding training period compensation amount. This can specifically solve the problem of unrepresentative training data caused by poor adversarial adaptation between the strategy and the optimal strategy. By compensating for the training period, the strategy can fully adapt in adversarial battles against the optimal strategy, collecting effective data that is more in line with actual adversarial scenarios, and providing sufficient and scientific data support for subsequent accurate determination of strategy effectiveness. Meanwhile, this invention dynamically updates the first threshold by summing two adjustment variables, achieving an organic fusion of the adjustment variables corresponding to the two core issues. This allows the final threshold adjustment to simultaneously meet the actual improvement needs of policy performance stability and adversarial adaptability, compensating for both performance decay and insufficient adversarial adaptability. This ensures that the adjusted first threshold accurately matches the current training state of the policy, providing sufficient and effective self-game training data for policy effectiveness determination. In normal scenarios where the policy has no performance decay and adversarial adaptability meets the standards, no additional processing is performed, maintaining the stability of the first threshold and avoiding unnecessary threshold adjustments that increase meaningless self-game computation, thus ensuring training efficiency. This allows the policy to complete effectiveness testing with sufficient data and accurate judgment, promoting efficient self-game training.Overall, this adjustment method employs a refined logical design that precisely identifies problem scenarios, uses a dedicated mapping table for quantitative matching, and integrates dual adjustment quantities to optimize stable thresholds in normal scenarios. This ensures that each adjustment of the initial threshold has a clear problem orientation and quantitative data support, achieving a threshold optimization mode where precise quantity supplementation is provided when problems exist and efficient judgment is made when no problems exist. This allows the adjusted initial threshold to accurately adapt to the actual combat effectiveness of the strategy and the needs of adversarial training. It effectively solves the problems of judgment distortion caused by insufficient data or inefficiency caused by data redundancy under fixed thresholds. This ensures that subsequent strategy effectiveness judgment can make objective and accurate judgments based on sufficient and effective training data. It provides scientific and reliable judgment threshold support for the closed loop of strategy training, effectiveness judgment, and strategy optimization / iteration in self-game training, further improving the accuracy and efficiency of the entire self-game training process. This promotes the steady progress of combat intelligent agent reinforcement learning training under the premise of data science and accurate judgment, continuously optimizing the comprehensive combat effectiveness of combat strategies.

[0038] Furthermore, if the cumulative number of self-play battles reaches a preset second threshold, and the current battle strategy is still judged to be invalid, then the battle strategy will be marked as an invalid strategy.

[0039] In this embodiment, the present invention constructs an ultimate rigid standard for determining invalid strategies and a training loss mitigation mechanism for reinforcement learning training of combat intelligent agents based on the self-game mode. Together with the dynamic determination and parameter adjustment mechanism based on the first threshold mentioned above, it forms a complete and bounded closed loop for strategy training. This completely breaks the drawback of conducting unrestricted self-game training and parameter adjustment for strategies with inherent shortcomings. From the three dimensions of training resource management, strategy library quality control, and training process standardization, it further improves the scientificity, efficiency, and controllability of the entire self-game training system. This invention is based on the premise that the cumulative number of self-game combat operations reaches a preset second threshold. This second threshold is the upper limit of the cumulative number of training operations after the strategy has undergone multiple rounds of self-game training based on the first threshold, and after dynamic adjustment of multiple dimensions such as discount factor, entropy regularization coefficient, and the ratio of the first threshold, it still fails to pass the effectiveness judgment. This means that the combat strategy has had sufficient opportunities for optimization attempts and has undergone multiple rounds of actual combat training and parameter adaptation adjustments, but it still cannot meet the effectiveness requirements under the current combat objectives. This is enough to prove that it has inherent shortcomings that are difficult to make up for through conventional adjustment and optimization, and does not have the optimization potential for further training or actual combat adaptation value. Under this premise, marking the current combat strategy as an invalid strategy can, on the one hand, promptly trigger the training loss mitigation mechanism, directly terminating all related operations such as subsequent self-game confrontation, dynamic parameter adjustment, and validity determination for the invalid strategy. This avoids continuously consuming valuable training resources such as computing resources, training time, and storage resources on strategies without optimization potential, significantly improving the utilization efficiency of training resources. It allows limited training resources to be tilted towards combat strategies with optimization potential and better aligned with combat objectives, ensuring the efficient advancement of overall self-game training from a resource perspective and effectively controlling training costs and training cycles. On the other hand, this operation also establishes clear invalid strategy removal criteria for the management of the combat strategy library. It can quickly mark and isolate such strategies without practical combat adaptability, preventing them from entering subsequent optimal strategy selection, strategy library iteration updates, and other stages. This prevents invalid strategies from interfering with the overall strategy optimization direction, purifying the combat strategy library from the source, and always maintaining the quality and adaptability of combat strategies within the strategy library to combat objectives, providing a high-quality strategy foundation for subsequent self-game training. Furthermore, the design of this invention enables the entire self-game training process of combat strategies to form a scientific logic of multiple optimization attempts and a final rigid judgment. It provides ample optimization opportunities for each combat strategy through the multi-round dynamic adjustment mentioned above, ensuring the flexibility of strategy optimization. At the same time, the final judgment of the second threshold defines the training boundary, avoiding bottomless trial-and-error training. This transforms the reinforcement learning training of combat agents from boundless strategy adjustment into a scientific training process that is systematic, bounded, controllable, and efficient in preventing losses.Overall, this invalid policy labeling step not only effectively solves the problem of invalid policies excessively consuming training resources, but also further standardizes the process boundaries of self-game training, improves the overall quality of the policy library, and makes the reinforcement learning training system of combat intelligent agents based on the self-game model more perfect, promoting the entire training process to always advance on a track of high efficiency, accuracy and practical combat needs.

[0040] like Figure 2 The diagram shows the structure of a reinforcement learning training system for combat agents based on a self-game model provided in this application embodiment. It includes: an optimal strategy selection module, a combat strategy evaluation module, and a combat strategy training module. The optimal strategy selection module dynamically acquires the priority weights of various tactical indicators based on the combat objective, comprehensively evaluates the tactical indicators of historical combat strategies based on these priority weights, and selects the current optimal strategy that matches the combat objective type based on the comprehensive evaluation results. The combat strategy evaluation module engages in self-game combat using the current combat strategy and the current optimal strategy, and generates a comprehensive strategy evaluation result for each combat operation in real time based on the agent's various tactical indicators and corresponding priority weights during the self-game process. The combat strategy training module dynamically determines the effectiveness of the current combat strategy based on the comprehensive strategy evaluation results within the first threshold number of self-game operations when the number of self-game operations reaches a preset first threshold number. If the current combat strategy is deemed effective, it replaces the current combat strategy with the optimal strategy; if the current combat strategy is deemed invalid, it dynamically adjusts the combat strategy based on the various tactical indicators and the comprehensive strategy evaluation results within the first threshold number of operations.

[0041] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)). Where there is no conflict, the solutions in the above embodiments can be used in combination.

[0042] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0043] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0044] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0045] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0046] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the scope and intent of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and variations.

Claims

1. A reinforcement learning training method for combat intelligent agents based on a self-game model, characterized in that, Includes the following steps: The priority weights of various tactical indicators are dynamically obtained based on the combat objectives. These tactical indicators include, but are not limited to, rewards received, casualty rate, troop loss ratio, and position retention rate. Based on the aforementioned priority weights, a comprehensive evaluation of various tactical indicators of historical combat strategies is conducted, and the current optimal strategy is selected based on the comprehensive evaluation results. The system engages in self-game combat using the current combat strategy and the current optimal strategy. During the self-game process, it generates a comprehensive strategy evaluation result for each combat operation in real time based on the agent's various tactical indicators and the corresponding priority weights. When the number of self-play battles reaches a preset first threshold, the effectiveness of the current battle strategy is dynamically determined based on the comprehensive strategy evaluation results within the first threshold. If the current battle strategy is determined to be effective, it is replaced with the optimal strategy. If the current battle strategy is determined to be ineffective, the battle strategy is dynamically adjusted based on various tactical indicators and comprehensive strategy evaluation results within the first threshold. The dynamic adjustment of the battle strategy means dynamically adjusting the entropy regularization coefficient and discount factor of the battle strategy to improve the learning efficiency and convergence of the battle strategy in subsequent self-play training. The entropy regularization coefficient is a hyperparameter used to adjust the balance between strategy exploration and utilization, and the discount factor is a hyperparameter used to weigh the importance of immediate rewards and future rewards.

2. The reinforcement learning training method for combat intelligent agents based on self-game mode as described in claim 1, characterized in that: The steps for dynamically acquiring the priority weights of various tactical indicators based on combat objectives include: Based on the current combat target type, the basic priority weights corresponding to each tactical indicator are matched from the preset indicator weight mapping relationship table. The indicator weight mapping relationship table is used to describe the correspondence between different combat target types and the initial weights of each tactical indicator. Based on current operational requirements, operational requirement parameters are dynamically quantified. These operational requirement parameters include, but are not limited to, mission timeliness urgency, troop redundancy parameters, equipment wear and tear threshold parameters, and tactical risk tolerance parameters. Based on the aforementioned operational requirement parameters, the adjustment components of each tactical indicator corresponding to each operational requirement parameter are matched from the preset indicator weight adjustment mapping relationship table, and all adjustment components of each tactical indicator are summed to obtain the dynamic adjustment amount of each tactical indicator. The indicator weight adjustment mapping relationship table is used to describe the correspondence between each operational requirement parameter and the adjustment components of each tactical indicator. Based on the current operational phase and tactical indicator type, determine the phase correction coefficient for each tactical indicator; wherein, if the tactical indicator is determined to be a key tactical indicator under the current operational phase, then assign it a strengthening correction coefficient preset for the current operational phase; if it is determined to be a non-key tactical indicator, then assign it a weakening correction coefficient preset for the current operational phase. For each tactical indicator, its basic priority weight is weighted and combined with the corresponding dynamic adjustment amount to obtain an intermediate weight value. Then, the intermediate weight value is multiplied by the stage correction coefficient to obtain the preliminary correction weight of the indicator. The initial revised weights of all tactical indicators are normalized to generate priority weights for each tactical indicator used in the current comprehensive assessment.

3. The reinforcement learning training method for combat intelligent agents based on self-game mode as described in claim 1, characterized in that: The steps of comprehensively evaluating various tactical indicators of historical combat strategies based on the priority weights, and selecting the current optimal strategy based on the comprehensive evaluation results, include: Select operational strategies from historical operational strategies that are compatible with the current operational objectives as preliminary operational strategies; Obtain preset tactical indicator evaluation thresholds, including but not limited to reward thresholds, casualty rate thresholds, troop loss ratio thresholds, and position retention rate thresholds; For each preliminary combat strategy, the actual values ​​of its various tactical indicators are compared with the corresponding tactical indicator evaluation thresholds to obtain the standard values ​​of each tactical indicator. Then, the standard values ​​of each tactical indicator are weighted and summed using the priority weights to obtain the comprehensive strategy evaluation index of the preliminary combat strategy. All preliminary operational strategies are ranked from highest to lowest according to their comprehensive strategic evaluation indicators, and the preliminary operational strategy ranked first is determined as the current optimal strategy.

4. The reinforcement learning training method for combat intelligent agents based on self-game mode as described in claim 1, characterized in that: The step of dynamically determining the effectiveness of the current combat strategy based on the comprehensive strategy evaluation results within the first threshold number includes: Step 1: Calculate the winning percentage of the current strategy in the self-play battles that reach the first threshold number of times, and determine whether the winning percentage exceeds the preset winning percentage threshold. If not, the current strategy is directly determined to be invalid. If so, proceed to Step 2. Step 2: Calculate the average achievement rate of the preset key tactical indicators in the self-game operation, and determine whether the average achievement rate of the key tactical indicators reaches the preset indicator achievement rate threshold. If not, the current operation strategy is determined to be invalid; if so, proceed to Step 3. Step 3: Based on the comprehensive strategy evaluation index of each battle in the self-game operation, calculate its decay rate with the order of battle, and determine whether the decay rate of the comprehensive evaluation index exceeds the preset index decay rate threshold. If so, the current battle strategy is determined to be invalid; otherwise, the current battle strategy is determined to be valid.

5. The reinforcement learning training method for combat intelligent agents based on self-game mode as described in claim 1, characterized in that: The steps for dynamically adjusting the operational strategy include: The discount factor is dynamically adjusted based on the average reward and average achievement rate of key tactical indicators for each combat operation within the first threshold number. Based on the comprehensive strategy evaluation index of each operation within the first threshold number, the entropy regularization coefficient is dynamically adjusted. Based on the decay rate of the comprehensive strategy evaluation index with the order of combat, and the adaptability measure of the current combat strategy to the current optimal strategy within the first number threshold, the first number threshold is dynamically adjusted.

6. The reinforcement learning training method for combat intelligent agents based on self-game mode as described in claim 5, characterized in that: The steps for dynamically adjusting the discount factor include: When the average reward is lower than the preset reward target and the average achievement rate of key tactical indicators does not reach the indicator achievement rate threshold, the first adjustment amount of the discount factor is obtained from the preset discount factor first adjustment mapping table based on the difference between the preset reward target and the average reward, and the second adjustment amount of the discount factor is obtained from the preset discount factor second adjustment mapping table based on the difference between the indicator achievement rate threshold and the average achievement rate of key tactical indicators, and the discount factor is dynamically reduced based on the first adjustment amount of the discount factor and the second adjustment amount of the discount factor. When the average reward is lower than the preset reward target and the average achievement rate of key tactical indicators reaches the indicator achievement rate threshold, the first adjustment amount of the discount factor is obtained from the preset discount factor first adjustment mapping relationship table based on the difference between the preset reward target and the average reward, and the discount factor is dynamically increased based on the first adjustment amount of the discount factor. When the average reward is not lower than the preset reward target and the average achievement rate of key tactical indicators does not reach the indicator achievement rate threshold, the second adjustment amount of the discount factor is obtained from the preset second adjustment mapping relationship table of discount factor based on the difference between the indicator achievement rate threshold and the average achievement rate of key tactical indicators, and the discount factor is dynamically increased based on the second adjustment amount of the discount factor. No additional processing is performed when the average reward received is not lower than the preset reward target and the average achievement rate of key tactical indicators reaches the indicator achievement rate threshold.

7. The reinforcement learning training method for combat intelligent agents based on self-game mode as described in claim 5, characterized in that: The step of dynamically adjusting the entropy regularization coefficient includes: Step 4: Determine whether the average comprehensive strategy evaluation index has reached the preset average comprehensive strategy evaluation threshold. If yes, proceed to step 5; otherwise, mark the current state as a state to be adjusted. Step 5: Determine whether the maximum value of the comprehensive strategy evaluation index has reached the preset maximum threshold for comprehensive strategy evaluation. If yes, proceed to Step 6; otherwise, mark the current state as a state to be adjusted. Step 6: Determine whether the minimum value of the comprehensive strategy evaluation index has reached the preset minimum threshold for comprehensive strategy evaluation. If yes, mark the current state as normal; otherwise, mark the current state as a state to be adjusted. If the current state is normal, no additional processing is required; If the current state is a state to be adjusted, then the entropy regularization coefficient is increased according to a preset incremental step size.

8. The reinforcement learning training method for combat intelligent agents based on self-game mode as described in claim 5, characterized in that, The step of dynamically adjusting the first threshold number includes: When the decay rate of the comprehensive strategy evaluation index with the order of combat exceeds the preset index decay rate threshold, or the adaptability metric is lower than the preset adaptability metric threshold, then based on the difference between the decay rate and the index decay rate threshold, the first adjustment amount of the threshold number is matched from the preset threshold number first adjustment mapping relationship table. The first adjustment mapping relationship table defines the correspondence between the decay rate deviation and the suggested training cycle extension amount. Based on the difference between the fitness metric threshold and the fitness metric, the second adjustment amount of the threshold number is obtained by matching from the preset threshold number second adjustment mapping table. The second adjustment mapping table defines the correspondence between the degree of fitness insufficiency and the suggested training cycle compensation amount. The summation operation is performed based on the first adjustment amount of the discount factor and the second adjustment amount of the discount factor, and the first number threshold is dynamically adjusted and updated based on the summation operation result; otherwise, no additional processing is performed.

9. The reinforcement learning training method for combat intelligent agents based on self-game mode as described in claim 1, characterized in that: If the cumulative number of self-play battles reaches a preset second threshold, and the current battle strategy is still determined to be invalid, then the battle strategy will be marked as an invalid strategy.

10. A combat agent reinforcement learning training system based on a self-game mode, applied in the combat agent reinforcement learning training method based on a self-game mode as described in any one of claims 1-9, characterized in that, include: Optimal strategy selection module, combat strategy evaluation module, and combat strategy training module; The optimal strategy selection module is used to dynamically obtain the priority weights of various tactical indicators based on the combat objective, comprehensively evaluate the various tactical indicators of historical combat strategies based on the priority weights, and select the current optimal strategy based on the comprehensive evaluation results. The combat strategy evaluation module is used to engage in self-game combat with the current combat strategy and the current optimal strategy, and in the self-game process, it generates a comprehensive strategy evaluation result for each combat operation in real time based on the various tactical indicators of the agent and the corresponding priority weights. The combat strategy training module is used to dynamically determine the effectiveness of the current combat strategy based on the comprehensive strategy evaluation results within the first threshold number of self-game combat attempts when the number of self-game combat attempts reaches a preset first threshold number of attempts. If the current combat strategy is determined to be effective, it is replaced with the optimal strategy. If the current combat strategy is determined to be invalid, the combat strategy is dynamically adjusted based on the various tactical indicators and comprehensive strategy evaluation results within the first threshold number of attempts.

Images