A reinforcement learning cross-simulation migration training method, device and equipment for complex land battle scenarios and a storage medium

Abstract

The application provides a reinforcement learning cross-simulation migration training method and device for a complex land battle scene, an equipment and a storage medium. The pre-training strategy network parameters are obtained by quickly pre-training a strategy network based on simulation interaction data in a lightweight land simulation environment. A cross-simulation state mapping module, an action space alignment module and a reward function consistency constraint module are constructed according to the differences in state space dimensions, action space granularity and reward value distribution between the two environments. The pre-training strategy network parameters are loaded into the strategy backbone network of the high-fidelity environment, and the state mapping module is connected to the input end of the strategy backbone network, the action alignment module is connected to the output end of the strategy backbone network, and the reward signal is corrected and adapted, so as to form a migration strategy network. Finally, the migration strategy network is fine-tuned in the high-fidelity land simulation environment, and the target decision strategy suitable for the high-fidelity battlefield environment is obtained.

Description

Technical Field

[0001] This invention relates to the field of interdisciplinary technology of artificial intelligence and military simulation, and in particular to a reinforcement learning cross-simulation transfer training method, apparatus, equipment and storage medium for complex land warfare scenarios. Background Technology

[0002] Deep reinforcement learning has garnered widespread attention in the field of military simulation due to its outstanding capabilities in complex sequential decision-making problems. Training agents capable of tactical decision-making, path planning, and multi-service coordination using deep reinforcement learning is an important direction in intelligent warfare research. A key approach to acquiring high-level intelligent decision-making strategies lies in training agents in a high-fidelity combat simulation environment with high-precision physical modeling, realistic geographic information support, and comprehensive military rule constraints.

[0003] However, the inherent characteristics of high-fidelity combat simulation environments mean that each step of the simulation—physical calculation, logical judgment, and damage assessment—consumes significant computational resources, resulting in a much slower single-step execution speed compared to simplified simulations. This creates a severe efficiency bottleneck for reinforcement learning agents undergoing complete training within such environments. For example, in a typical land combat scenario, an agent needs to complete tens of thousands of training sessions to achieve initial convergence. The time consumed per session and limited speedup of a high-fidelity simulation platform mean that completing this training workload could take hundreds of days, making effective strategy iteration difficult. A natural approach is to first perform basic strategy training in a lightweight simulation environment, which is hundreds of times faster, and then transfer the pre-trained strategy to the high-fidelity environment for fine-tuning. However, this approach faces a core technical obstacle in practice: lightweight and high-fidelity simulation environments are heterogeneous systems. They exhibit systematic differences in the dimension and semantics of the state space, the granularity and interface of the action space, and the numerical deviations in reward values ​​due to differences in the underlying physical models under the same reward function definition. Therefore, pre-trained strategies cannot be effectively run directly in the high-fidelity environment.

[0004] Existing technologies have proposed several solutions for policy transfer between simulation environments, but none have systematically solved the aforementioned heterogeneous alignment problem. Course learning methods design a sequence of tasks from easy to difficult within a single simulation environment, without addressing cross-environment transfer and failing to leverage the efficiency advantages of lightweight environments. Domain randomization methods enhance policy robustness by randomizing simulation parameters, but only handle parameter variations within the same environment, failing to bridge the fundamental differences between two heterogeneous systems in state representation structure, action interface definition, and dynamic model. Simulation-to-reality transfer methods focus on domain adaptation between simulation and the real physical world, addressing the distribution offset of continuous physical quantities, rather than the structural alignment problems of mismatched state space dimensions, asymmetrical action semantics, and inconsistent reward values ​​between two digital simulation systems.

[0005] In view of the above, this application is hereby submitted. Summary of the Invention

[0006] This invention discloses a reinforcement learning cross-simulation transfer training method, device, equipment, and storage medium for complex land warfare scenarios. It aims to solve the problem that in complex land warfare scenarios, the pre-trained strategy cannot be directly transferred to the high-fidelity environment for effective operation due to the systematic differences in state space dimension, action space granularity, and reward value distribution between lightweight simulation environments and high-fidelity simulation environments, thus failing to balance training efficiency and final strategy fidelity.

[0007] The first embodiment of the present invention provides a reinforcement learning cross-simulation transfer training method for complex land warfare scenarios, applicable to pre-constructed lightweight land warfare simulation environments and high-fidelity land warfare simulation environments, including: The task configuration of the land warfare scenario is input into the lightweight land warfare simulation environment, which generates simulation interaction data. Based on the simulation interaction data, the policy network is iteratively optimized and trained using a reinforcement learning algorithm until convergence, and the pre-trained policy network parameters are output. To address the differences between the lightweight land combat simulation environment and the high-fidelity land combat simulation environment in terms of state space dimension, action space granularity, and reward value distribution, a cross-simulation state mapping module, an action space alignment module, and a reward function consistency constraint module are constructed. The pre-trained policy network parameters are loaded into the policy backbone network of the high-fidelity land combat simulation environment. The cross-simulation state mapping module is connected to the input of the policy backbone network, and the action space alignment module is connected to the output of the policy backbone network. Simultaneously, the reward signal is corrected and adapted based on the reward function consistency constraint module, and a transfer policy network is output. The migration strategy network is loaded into the high-fidelity land warfare simulation environment, and high-fidelity simulation interaction data is generated by the high-fidelity land warfare simulation environment. The migration strategy network is continuously fine-tuned and trained based on the high-fidelity simulation interaction data to output a target decision strategy that adapts to the high-fidelity battlefield environment.

[0008] The second embodiment of the present invention provides a reinforcement learning cross-simulation transfer training device for complex land warfare scenarios, applicable to pre-constructed lightweight land warfare simulation environments and high-fidelity land warfare simulation environments, including: The strategy pre-training unit is used to input the mission configuration of the land warfare scenario into the lightweight land warfare simulation environment, generate simulation interaction data from the lightweight land warfare simulation environment, and use reinforcement learning algorithm to iteratively optimize and train the strategy network based on the simulation interaction data until convergence, and output the pre-trained strategy network parameters. The cross-simulation transfer alignment unit is used to address the differences between the lightweight land combat simulation environment and the high-fidelity land combat simulation environment in terms of state space dimension, action space granularity, and reward value distribution. It constructs a cross-simulation state mapping module, an action space alignment module, and a reward function consistency constraint module. The pre-trained policy network parameters are loaded into the policy backbone network of the high-fidelity land combat simulation environment. The cross-simulation state mapping module is connected to the input of the policy backbone network, and the action space alignment module is connected to the output of the policy backbone network. Simultaneously, the reward signal is corrected and adapted based on the reward function consistency constraint module, and a transfer policy network is output. The high-fidelity environment fine-tuning unit is used to load the migration strategy network into the high-fidelity land combat simulation environment, generate high-fidelity simulation interaction data from the high-fidelity land combat simulation environment, and continuously perform reinforcement learning fine-tuning training on the migration strategy network based on the high-fidelity simulation interaction data to output a target decision strategy adapted to the high-fidelity battlefield environment.

[0009] The third embodiment of the present invention provides a reinforcement learning cross-simulation transfer training device for complex land warfare scenarios, characterized in that it includes a memory and a processor, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement a reinforcement learning cross-simulation transfer training method for complex land warfare scenarios as described in any of the above embodiments.

[0010] The fourth embodiment of the present invention provides a computer-readable storage medium storing a computer program, which can be executed by the processor of the device where the computer-readable storage medium is located, to implement a reinforcement learning cross-simulation transfer training method for complex land warfare scenarios as described in any of the above embodiments.

[0011] Based on the reinforcement learning cross-simulation transfer training method, apparatus, device, and storage medium provided by this invention for complex land warfare scenarios, the method first rapidly pre-trains the policy network based on simulation interaction data in a lightweight land warfare simulation environment to obtain pre-trained policy network parameters. Then, considering the differences in state space dimension, action space granularity, and reward value distribution between the two environments, a cross-simulation state mapping module, an action space alignment module, and a reward function consistency constraint module are constructed. The pre-trained policy network parameters are loaded into the policy backbone network of a high-fidelity environment, and the state mapping module is connected to its input end, the action alignment module is connected to its output end, and the reward signal is corrected and adapted to form a transfer policy network. Finally, the transfer policy network is fine-tuned and trained in a high-fidelity land warfare simulation environment. This method retains the advantages of efficient training in a lightweight environment while systematically bridging the structural differences between heterogeneous environments, thereby obtaining a target decision-making strategy adapted to the high-fidelity battlefield environment. Attached Figure Description

[0012] Figure 1 This is a flowchart illustrating a reinforcement learning cross-simulation transfer training method for complex land warfare scenarios provided in the first embodiment of the present invention. Figure 2 This is a schematic diagram of a module of a reinforcement learning cross-simulation transfer training device for complex land warfare scenarios provided in the second embodiment of the present invention. Detailed Implementation

[0013] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0014] To better understand the technical solution of the present invention, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0015] This invention discloses a reinforcement learning cross-simulation transfer training method, device, equipment, and storage medium for complex land warfare scenarios. It aims to solve the problem that in complex land warfare scenarios, the pre-trained strategy cannot be directly transferred to the high-fidelity environment for effective operation due to the systematic differences in state space dimension, action space granularity, and reward value distribution between lightweight simulation environments and high-fidelity simulation environments, thus failing to balance training efficiency and final strategy fidelity.

[0016] Please see Figure 1 The first embodiment of this invention provides a reinforcement learning cross-simulation transfer training method for complex land warfare scenarios, applied to a pre-constructed lightweight land warfare simulation environment and a high-fidelity land warfare simulation environment. The lightweight land warfare simulation environment employs an abstract physical model and rapid rule deduction based on probability and tables, while the high-fidelity land warfare simulation environment features high-precision physical and behavioral modeling, support from real geographic information systems, and comprehensive military rule constraints. The method includes the following steps: S101, Input the mission configuration of the land warfare scenario into the lightweight land warfare simulation environment, generate simulation interaction data from the lightweight land warfare simulation environment, use reinforcement learning algorithm to iteratively optimize and train the policy network based on the simulation interaction data until convergence, and output the pre-trained policy network parameters. It should be noted that in the strategy pre-training step, a land combat scenario involving several main battle tanks of the Red Force against several main battle tanks, several transport vehicles, and several infantry of the Blue Force is input into a lightweight land combat simulation environment. This environment uses a simplified continuous dynamics model to drive entity movement, ignoring complex effects such as track-terrain interaction. Collision detection is simplified to a bounding box method, and hit determination and damage calculation in the combat rules use fast algorithms based on probability and tables. The simulation speed for a single game can reach tens to hundreds of times that of a high-fidelity environment.

[0017] The state space of this environment is represented by structured low-dimensional feature vectors. Entity state features include unit type, name, two-dimensional plane coordinates, health points, main weapon information, and enemy units within line of sight. Battlefield situation features include distance to friendly forces and distance to all enemy forces. Mission context features include the number of surviving friendly units, the number of destroyed friendly units, the number of surviving enemy units, and the number of destroyed enemy units. The action space uses a discretized high-level tactical command set. The movement distance of maneuver commands is related to the simulation step size and entity speed, and the speed increases and decreases abruptly without acceleration or deceleration. Reverse maneuvers directly move to the rear target point without performing a turn-around action. Attack commands contain friendly entity IDs and enemy target IDs, and their ammunition type field is filled with fixed default values ​​as placeholders. The actual ammunition type executed is forcibly overridden by the fixed rule engine within the environment during the command parsing phase. The decision space of the policy network does not include an ammunition selection dimension. The reward function adopts a hybrid reward mechanism. Sparse task rewards include single-round victory rewards, single-round defeat rewards, single-round combat time rewards upon victory, and damage rewards upon victory. Dense process rewards include rewards for inflicting damage on the enemy, rewards for destroying enemy units, penalties for friendly units being damaged, penalties for friendly units being destroyed, rewards for capturing key positions, and rewards for reconnaissance time. Among them, dense process rewards are fed back in real time at each simulation time step to guide the policy network to obtain continuous learning gradients in addition to sparse win / loss signals.

[0018] The lightweight land warfare simulation environment and the high-fidelity land warfare simulation environment employ the same reinforcement learning algorithm (such as MAPPO or MADDPG) and the same policy backbone network architecture. That is, the backbone network after the feature extraction layer is completely identical to the decision head structure, with only the input layer dimension differing due to their respective state spaces. This allows pre-trained converged network weights to be completely transferred to the high-fidelity environment through element-wise replication, without information loss introduced by structural adaptation. Furthermore, hyperparameter configurations can be directly reused, avoiding the need for time-consuming hyperparameter tuning in the high-fidelity environment.

[0019] The training process employs a multi-instance parallel sampling mechanism, simultaneously launching N parallel lightweight simulation instances on a single server. Each instance independently runs adversarial tasks and generates its own simulation interaction data. All datasets are centrally stored in a shared experience pool. The trainer randomly samples batches of data from this pool to update the policy network parameters, increasing the amount of experience collected per unit time by N times. The training speed grows approximately linearly with the number of parallel instances. With this configuration, the policy network achieves initial convergence in a lightweight environment after large-scale interactive sampling and iterative optimization, learning basic tactical behaviors such as utilizing terrain cover, rapidly maneuvering to enemy territory, and distinguishing high-value targets. The entire pre-training process can be completed within 2 to 3 days, outputting the pre-trained policy network parameters for subsequent cross-simulation transfer alignment steps.

[0020] S102, addressing the differences between the lightweight land combat simulation environment and the high-fidelity land combat simulation environment in terms of state space dimension, action space granularity, and reward value distribution, a cross-simulation state mapping module, an action space alignment module, and a reward function consistency constraint module are constructed; the pre-trained policy network parameters are loaded into the policy backbone network of the high-fidelity land combat simulation environment, and the cross-simulation state mapping module is connected to the input end of the policy backbone network, and the action space alignment module is connected to the output end of the policy backbone network. Simultaneously, the reward signal is corrected and adapted based on the reward function consistency constraint module, and a transfer policy network is output. In the cross-simulation transfer alignment step, in view of the systematic differences between the lightweight land combat simulation environment and the high-fidelity land combat simulation environment in terms of state space dimension, action space granularity and reward value distribution, a cross-simulation state mapping module, an action space alignment module and a reward function consistency constraint module are constructed respectively. The pre-trained policy network parameters are loaded into the policy backbone network of the high-fidelity environment and connected to each alignment module to form a transfer policy network.

[0021] The cross-simulation state mapping module is constructed as follows. Let the state space dimension of the lightweight simulation environment be M, and the state space dimension of the high-fidelity simulation environment be N, typically M less than N. A learnable linear projection layer is added to the input of the high-fidelity environment policy network. This projection layer is a single-layer fully connected network Linear(N, M) without activation functions, and its mapping function is S_mapped = W·S_high + b, where W is a learnable weight matrix of dimension M×N, b is a learnable bias vector of dimension M, S_high is the original high-dimensional state vector output by the high-fidelity environment, and S_mapped is the mapped low-dimensional state vector. S_mapped is passed as input to the policy backbone network loaded from the pre-trained policy network parameters, enabling the backbone network to directly process the high-dimensional states of the high-fidelity environment. The projection layer has only M×N parameters, making it less prone to overfitting, while achieving end-to-end automatic mapping learning, fully reusing backbone network parameters, and maximizing knowledge retention.

[0022] The training of the linear projection layer is divided into two phases. In the frozen training phase, all parameters of the policy backbone network loaded from the pre-trained policy network parameters are frozen. Only the weight matrix W and bias vector b of the linear projection layer are trained, with the training loss function being L = MSE(π_frozen(S_mapped), a_ref), where π_frozen is the frozen policy backbone network parameters, and a_ref is the reference action output by the reference policy under the same situation. This loss function forces the linear projection layer to learn a mapping relationship, enabling the compressed high-fidelity state S_high, when input into the pre-trained policy network, to produce an output consistent with the reference action. In the joint training phase, the policy backbone network parameters are unfrozen, and the linear projection layer and the policy backbone network are jointly trained end-to-end, allowing the entire network to be co-optimized under the real-world state distribution in a high-fidelity environment.

[0023] The action space alignment module comprises a maneuver command alignment submodule and a strike command alignment submodule. The maneuver command alignment submodule maps discrete maneuver distances in the lightweight environment to continuous maneuver distances in the high-fidelity environment through coordinate scale mapping. The maneuver command format is identical in both environments; the only difference lies in whether the distance parameter is discrete or continuous. The core problem that the strike command alignment submodule needs to solve is that in the lightweight environment, the policy network never learns ammunition selection, and the ammunition type field in the strike command is always covered by the environment rule engine. However, the high-fidelity environment requires the policy network to autonomously output valid ammunition identifiers, and ammunition selection directly affects the hit probability and damage effect. To address this, the strike command alignment submodule employs a three-stage mechanism: rule-anchored initialization, capability dimension expansion, and progressive annealing takeover.

[0024] During the rule-anchoring initialization phase, when the migration strategy network first runs in a high-fidelity environment, the rule anchor takes over the ammunition decision-making. This rule anchor uses target type, engagement distance, and target attitude as input variables and outputs the corresponding ammunition type according to a pre-set military expert rule base. Specifically, the rule base encodes expert experience and equipment operational doctrine into executable decision logic. For example, when facing armored targets head-on, kinetic energy munitions are prioritized at close range to counter composite armor; chemical energy munitions are selected to improve damage efficiency when facing armored targets exposed to the sides or rear; guided munitions are selected to ensure hit probability when facing long-range targets; and appropriate lethal munitions are selected based on the degree of exposure and protection level when facing soft targets and fortified targets. This phase ensures that the strategy network can execute complete and tactically sound strike commands from the first frame, establishing a stable value baseline for subsequent learning.

[0025] In the capability dimension expansion phase, an ammunition selection head is added in series at the end of the policy backbone network. The ammunition selection head takes the situation feature vector output by the policy backbone network as input, directly reusing the high-level situation features already calculated by the backbone network without re-extracting features, achieving computational efficiency and semantic alignment. This ammunition selection head adopts a lightweight design with a three-layer fully connected network. The first layer includes a linear layer, a ReLU activation function, and a Dropout layer (with a dropout rate of 0.1), compressing the situation features and introducing non-linear expressive power. The second layer includes a linear layer and a ReLU activation function to further refine high-order ammunition decision features. The third layer is a linear output layer that maps the features to the ammunition category space, outputting unnormalized logits. Finally, the logits are converted into a probability distribution for all ammunition categories using a Softmax function. The entire ammunition selection head has no more than 2000 parameters, uses random initialization to learn ammunition selection from scratch, and is decoupled from the backbone network, allowing the backbone to be frozen or fine-tuned while the ammunition head remains trainable.

[0026] In the progressive annealing takeover phase, the rule dependency coefficient λ(t) is controlled by the annealing scheduling function to achieve a smooth transition from rule anchoring to policy autonomy. Let T_total be the total number of steps for fine-tuning training in the high-fidelity environment, T_warmup be the number of warm-up steps and T_warmup = 0.05 × T_total, and T_anneal be the annealing period and T_anneal = 0.4 × T_total. When t is less than T_warmup, λ(t) = 1.0, meaning the policy network fully relies on rules to adapt to the basic interaction interface of the high-fidelity environment. When t is not less than T_warmup, λ(t) = max(0, 1 - (t - T_warmup) / T_anneal) decays linearly. During the annealing cycle, for a given current state s and target information target_info, the final ammunition selection probability P_final is a weighted fusion of the output probability P_rule of the rule anchor and the output probability P_network of the ammunition selection head, P_final = λ(t)·P_rule + (1-λ(t))·P_network, where s is the current battlefield state output by the high-fidelity environment, target_info is the type, distance, and attitude information of the current target, P_rule is the ammunition type probability output by the rule anchor based on target_info, and P_network is the ammunition category probability distribution output by the ammunition selection head based on s, sampled from P_final during execution. When t is not less than T_warmup + T_anneal, λ(t) decays to zero, the rule anchor completely exits, and ammunition selection is entirely decided autonomously by the ammunition selection head. At this point, the policy network has fully acquired ammunition optimization capabilities.

[0027] The reward function consistency constraint module employs a three-level progressive constraint mechanism. Both environments use identical reward function definitions, and are completely consistent in reward function structure, weighting coefficients, and sparsity design. However, differences in the fidelity of the underlying physical model, sensor simulation, and damage calculation lead to different numerical reward feedback for the same state-action pairs. This is the core bias problem that needs to be addressed.

[0028] In the offline bias prior modeling stage, initial states covering various terrains and enemy / friendly situations are randomly sampled in a lightweight environment. A hybrid strategy is used to generate action sequences, with 40% random strategies used to explore state space coverage, 40% rule-based strategies used to cover tactical-related areas, and 20% pre-trained early strategies used to cover the actual distribution. The generated state-action sequences are executed in parallel in both the lightweight and high-fidelity environments. The state s_t, action a_t, and rewards returned by the two environments are recorded at each time step t. The reward difference ΔR_t = R_high^t - R_low^t is calculated, where R_high^t is the reward returned by the high-fidelity environment at time step t, R_low^t is the reward returned by the lightweight environment at time step t, s_t is the state at time step t, and a_t is the action executed at time step t. This constructs the reward difference dataset D = {(s_t, a_t, ΔR_t)}. Based on this dataset, a bias prediction network is trained, enabling it to predict the corresponding reward value bias in a high-fidelity environment based on the state-action pair of a lightweight environment.

[0029] During the pre-training reward correction phase, the bias prediction network is used online to correct the original reward returned by the lightweight environment during the training of the policy network. After each time step t, the correction reward R_corrected = R_raw + α(t)·ΔR_pred is calculated, where R_raw is the original reward, ΔR_pred is the bias prediction value output by the bias prediction network, and α(t) is the correction strength coefficient. To prevent inaccurate bias prediction in the early stages from causing training instability, α(t) adopts a progressive scheduling strategy. When t is less than T_warmup, α(t) = 0, which is the warm-up period and completely depends on the original reward. When T_warmup is not greater than t and t is less than T_warmup + T_ramp, α(t) = α_max·(t - T_warmup) / T_ramp, which is the progressive period and the correction strength increases linearly. When t is not less than T_warmup + T_ramp, α(t) = α_max, where α_max is the maximum correction strength. The value network is also trained based on the corrected reward, which allows the policy network to adapt to the real reward distribution of the high-fidelity environment in advance in a lightweight environment, avoiding the learning of ineffective policies from the source and outputting corrected value network parameters.

[0030] During the transfer value adaptation phase, the parameters of the corrected value network are loaded as the corrected value network V_corrected during fine-tuning training. Simultaneously, the high-fidelity value network V_high is randomly initialized and trained based on the real rewards returned by the high-fidelity environment. The temporal difference advantage estimate A_corrected = R_high + γ·V_corrected(s') - V_corrected(s) for the corrected value network and the temporal difference advantage estimate A_high = R_high + γ·V_high(s') - V_high(s) for the high-fidelity value network are calculated respectively, where γ is the discount factor and s' is the state at the next time step. Policy updates are performed using a progressively fused advantage function A_final = β(t)·A_corrected + (1 - β(t))·A_high, with the policy loss being L_policy = -E[log π(a|s)·A_final], where β(t) is the fusion coefficient and β(t) = max(0,1 - t / T_adapt) decays from 1 to 0 with each fine-tuning step, and T_adapt is the adaptation period. The action-aligned annealing mechanism and the value-adapted annealing mechanism operate in parallel, achieving a synchronous and smooth transfer of decision-making ability and value perception. When β(t) decays to zero, the policy update relies entirely on the advantage estimation of the high-fidelity value network. The transfer policy network completes a smooth transition from corrected value to true value under the drive of the real reward signal in the high-fidelity environment.

[0031] S103, the migration strategy network is loaded into the high-fidelity land combat simulation environment, high-fidelity simulation interaction data is generated by the high-fidelity land combat simulation environment, and the migration strategy network is continuously fine-tuned and trained based on the high-fidelity simulation interaction data to output a target decision strategy that adapts to the high-fidelity battlefield environment.

[0032] In the high-fidelity environment fine-tuning step, the migration strategy network output from the cross-simulation migration alignment step is loaded into the high-fidelity land warfare simulation environment for operation. This migration strategy network includes the strategy backbone network and value network fully inherited from the pre-training stage, as well as the linear projection layer, ammunition selection head, and correction value network added in the migration alignment step. The parameters of the strategy backbone network and value network are fully loaded from the pre-trained strategy network parameters in an element-wise copy manner, while the linear projection layer and ammunition selection head are randomly initialized.

[0033] In a high-fidelity environment, the agent's operational flow is as follows: At each simulation time step, the high-fidelity environment outputs a high-dimensional raw observation S_high. This observation is mapped to a low-dimensional state vector S_mapped by a linear projection layer and then input into the policy backbone network. The policy backbone network extracts situational features based on S_mapped and outputs maneuver commands and target selection. Simultaneously, the ammunition selection head receives the situational feature vector output by the policy backbone network and generates an ammunition category probability distribution. During the annealing cycle, this probability distribution is weighted and fused with the output of the rule anchor to generate a complete strike command. After receiving and executing the above command, the high-fidelity environment returns a reward R_high calculated based on real combat rules.

[0034] Fine-tuning training employs a hierarchical learning rate strategy to balance knowledge retention and adaptation to new capabilities. The linear projection layer and the ammunition selection head, as new modules, use a relatively high first learning rate (e.g., 3e-4) to enable them to quickly learn state mappings and ammunition decision-making capabilities. The value network inherited from the pre-training stage uses a moderate second learning rate (e.g., 1e-4) to adapt to the real reward distribution in a high-fidelity environment while retaining existing value knowledge. The policy backbone network, also inherited from the pre-training stage, uses the lowest third learning rate (e.g., 5e-6) for only fine-tuning to retain basic tactical knowledge learned in a lightweight environment. The first learning rate is greater than the second, and the second is greater than the third. This gradient difference in the three learning rates ensures that new modules can quickly acquire new capabilities, while the mature knowledge of existing modules is not rapidly overwritten, avoiding catastrophic forgetting and policy oscillations.

[0035] During the fine-tuning process, the various annealing mechanisms designed in the aforementioned cross-simulation transfer alignment steps operate collaboratively. In the action alignment dimension, the rule dependency coefficient λ(t) for ammunition selection linearly decays from 1.0 to zero according to the annealing scheduling function, allowing the policy network to progressively acquire complete autonomous ammunition decision-making capabilities. In the reward alignment dimension, the fusion coefficient β(t) of the corrected value network V_corrected and the high-fidelity value network V_high synchronously decays from 1 to zero, smoothly transitioning the value baseline of the policy update from relying on the corrected value estimate to fully relying on the high-fidelity true value estimate. The parallel execution of the two annealing processes ensures that the expansion of decision-making capabilities and the adaptation of value cognition are completed synchronously over time. Driven by the continuous real reward signals in the high-fidelity environment, all network modules collaboratively optimize throughout the fine-tuning process, and the transfer policy network gradually evolves from the optimal state in the lightweight environment to the optimal state in the high-fidelity environment.

[0036] Through the synergistic effect of the aforementioned stratified learning rate and progressive annealing, the number of fine-tuning training rounds required in the high-fidelity environment is reduced from tens of thousands of rounds required for direct training to hundreds of rounds. The overall training cycle is shortened by one to two orders of magnitude compared to completing all training directly in the high-fidelity environment. The final target decision-making strategy is effective, adaptable, and robust under the constraints of the complete physical model, complex terrain dynamics, and multi-service coordination in the high-fidelity environment.

[0037] Please see Figure 2 The second embodiment of the present invention provides a reinforcement learning cross-simulation transfer training device for complex land warfare scenarios, applicable to pre-constructed lightweight land warfare simulation environments and high-fidelity land warfare simulation environments, including: The strategy pre-training unit 201 is used to input the task configuration of the land warfare scenario into the lightweight land warfare simulation environment, generate simulation interaction data from the lightweight land warfare simulation environment, use reinforcement learning algorithm to iteratively optimize and train the policy network based on the simulation interaction data until convergence, and output the pre-trained policy network parameters. The cross-simulation transfer alignment unit 202 is used to address the differences between the lightweight land combat simulation environment and the high-fidelity land combat simulation environment in terms of state space dimension, action space granularity, and reward value distribution. It constructs a cross-simulation state mapping module, an action space alignment module, and a reward function consistency constraint module. The pre-trained policy network parameters are loaded into the policy backbone network of the high-fidelity land combat simulation environment. The cross-simulation state mapping module is connected to the input of the policy backbone network, the action space alignment module is connected to the output of the policy backbone network, and the reward signal is corrected and adapted based on the reward function consistency constraint module to output the transfer policy network. The high-fidelity environment fine-tuning unit 203 is used to load the migration strategy network into the high-fidelity land combat simulation environment, generate high-fidelity simulation interaction data from the high-fidelity land combat simulation environment, and continuously perform reinforcement learning fine-tuning training on the migration strategy network based on the high-fidelity simulation interaction data to output a target decision strategy adapted to the high-fidelity battlefield environment.

[0038] The third embodiment of the present invention provides a reinforcement learning cross-simulation transfer training device for complex land warfare scenarios, characterized in that it includes a memory and a processor, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement a reinforcement learning cross-simulation transfer training method for complex land warfare scenarios as described in any of the above embodiments.

[0039] The fourth embodiment of the present invention provides a computer-readable storage medium storing a computer program, which can be executed by the processor of the device where the computer-readable storage medium is located, to implement a reinforcement learning cross-simulation transfer training method for complex land warfare scenarios as described in any of the above embodiments.

[0040] Based on the reinforcement learning cross-simulation transfer training method, apparatus, device, and storage medium provided by this invention for complex land warfare scenarios, the method first rapidly pre-trains the policy network based on simulation interaction data in a lightweight land warfare simulation environment to obtain pre-trained policy network parameters. Then, considering the differences in state space dimension, action space granularity, and reward value distribution between the two environments, a cross-simulation state mapping module, an action space alignment module, and a reward function consistency constraint module are constructed. The pre-trained policy network parameters are loaded into the policy backbone network of a high-fidelity environment, and the state mapping module is connected to its input end, the action alignment module is connected to its output end, and the reward signal is corrected and adapted to form a transfer policy network. Finally, the transfer policy network is fine-tuned and trained in a high-fidelity land warfare simulation environment. This method retains the advantages of efficient training in a lightweight environment while systematically bridging the structural differences between heterogeneous environments, thereby obtaining a target decision-making strategy adapted to the high-fidelity battlefield environment.

[0041] Exemplary examples show that the computer program described in the third and fourth embodiments of the present invention can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in implementing a reinforcement learning cross-simulation transfer training device for complex land warfare scenarios. For example, the apparatus described in the second embodiment of the present invention.

[0042] The processor referred to can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. This processor is the control center of the reinforcement learning cross-simulation transfer training method for complex land warfare scenarios, connecting various parts of the method through various interfaces and lines.

[0043] The memory can be used to store the computer programs and / or modules. The processor, by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory, implements various functions of a reinforcement learning cross-simulation transfer training method for complex land warfare scenarios. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, text conversion function, etc.), etc.; the data storage area may store data created based on the use of the mobile phone (such as audio data, text message data, etc.). In addition, the memory may include high-speed random access memory and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0044] If the implemented module is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0045] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0046] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A reinforcement learning cross-simulation transfer training method for complex land warfare scenarios, applied to pre-constructed lightweight land warfare simulation environments and high-fidelity land warfare simulation environments, characterized in that... include: The task configuration of the land warfare scenario is input into the lightweight land warfare simulation environment, which generates simulation interaction data. Based on the simulation interaction data, the policy network is iteratively optimized and trained using a reinforcement learning algorithm until convergence, and the pre-trained policy network parameters are output. To address the differences between the lightweight land combat simulation environment and the high-fidelity land combat simulation environment in terms of state space dimension, action space granularity, and reward value distribution, a cross-simulation state mapping module, an action space alignment module, and a reward function consistency constraint module are constructed. The pre-trained policy network parameters are loaded into the policy backbone network of the high-fidelity land combat simulation environment. The cross-simulation state mapping module is connected to the input end of the policy backbone network, and the action space alignment module is connected to the output end of the policy backbone network. At the same time, the reward signal is corrected and adapted based on the reward function consistency constraint module, and the transfer policy network is output. The reward function consistency constraint module employs a three-level progressive constraint mechanism, including offline bias prior modeling, pre-training reward correction, and transfer value adaptation. The offline bias prior modeling is as follows: Initial states covering various terrains and enemy / friendly situations are randomly sampled in a lightweight environment. A hybrid strategy is used to generate action sequences. The generated state-action sequences are executed in parallel in both the lightweight and high-fidelity environments. The state s_n, action a_n, and rewards returned by the two environments are recorded at each time step n. The reward difference ΔR_n = R_high^n - R_low^n is calculated, where R_high^n is the reward returned by the high-fidelity land combat simulation environment at time step n, and R_low^n is the reward returned by the lightweight land combat simulation environment at time step n. A reward difference dataset D = {(s_n, a_n, ΔR_n)} is constructed based on this dataset. A bias prediction network is trained on this dataset, and the bias prediction network outputs a predicted bias value for the reward value of a given state-action pair. The pre-training reward correction is as follows: during the training of the policy network, the deviation prediction network is used online to correct the original reward returned by the lightweight land combat simulation environment. The correction formula is R_corrected = R_raw + α(k)·ΔR_pred, where R_raw is the original reward, ΔR_pred is the deviation prediction value output by the deviation prediction network, and α(k) is the correction strength coefficient. Based on the corrected reward R_corrected, the value network is trained synchronously, and the corrected value network parameters are output. The migration value adaptation is as follows: the parameters of the corrected value network are loaded into the corrected value network V_corrected, and the high-fidelity value network V_high is randomly initialized and trained based on the real rewards returned by the high-fidelity land combat simulation environment. The temporal differential advantage estimate A_corrected of the corrected value network and the temporal differential advantage estimate A_high of the high-fidelity value network are calculated respectively. The policy is updated by progressively fusing the advantage function A_final = β(h)·A_corrected+ (1 - β(h))·A_high, where β(h) is the fusion coefficient and decays from 1 to 0 with the number of fine-tuning steps, β(h) = max(0, 1 - h / T_adapt), and T_adapt is the adaptation period. The migration strategy network is loaded into the high-fidelity land warfare simulation environment, and high-fidelity simulation interaction data is generated by the high-fidelity land warfare simulation environment. The migration strategy network is continuously fine-tuned and trained based on the high-fidelity simulation interaction data to output a target decision strategy that adapts to the high-fidelity battlefield environment.

2. The reinforcement learning cross-simulation transfer training method for complex land warfare scenarios according to claim 1, characterized in that, The cross-simulation state mapping module is constructed as follows: A learnable linear projection layer is added to the input of the strategy backbone network of the high-fidelity land warfare simulation environment. The linear projection layer is a single-layer fully connected network Linear(N, M) without activation function, where N is the state space dimension of the high-fidelity land warfare simulation environment, M is the state space dimension of the lightweight land warfare simulation environment, and M is less than N. The mapping function of the linear projection layer is S_mapped = W·S_high + b, where W is a learnable weight matrix of dimension M×N, b is a learnable bias vector of dimension M, S_high is the original high-dimensional state vector output by the high-fidelity land combat simulation environment, and S_mapped is the mapped low-dimensional state vector. The S_mapped is passed as input to the policy backbone network loaded from the pre-trained policy network parameters; The training of the linear projection layer is divided into two stages: Freeze training phase: Freeze the parameters of the policy backbone network loaded by the pre-trained policy network parameters, and train only the weight matrix W and bias vector b of the linear projection layer. The training loss function is L = MSE(π_frozen(S_mapped),a_ref), where π_frozen is the policy backbone network with frozen parameters, and a_ref is the reference action output by the reference policy under the same situation. Joint training phase: Unfreeze the parameters of the policy backbone network and perform end-to-end joint training of the linear projection layer and the policy backbone network.

3. The reinforcement learning cross-simulation transfer training method for complex land warfare scenarios according to claim 1, characterized in that, The motion space alignment module includes a maneuver command alignment submodule and a strike command alignment submodule: The maneuver command alignment submodule maps the discrete maneuver distance in the lightweight land combat simulation environment to the continuous maneuver distance in the high-fidelity land combat simulation environment through coordinate scale mapping. The strike instruction alignment sub-module adopts a three-stage mechanism of rule-anchored initialization, ability dimension expansion, and progressive annealing takeover, enabling the pre-trained policy network without the ammunition selection dimension to obtain the ammunition selection decision-making ability in the high-fidelity land warfare simulation environment.

4. The reinforcement learning cross-simulation transfer training method for complex land warfare scenarios according to claim 3, characterized in that, The three-stage mechanism of the strike instruction alignment sub-module is specifically as follows: Rule-anchored initialization stage: When the migration policy network first runs in the high-fidelity land warfare simulation environment, the rule anchor takes over ammunition decision-making. The rule anchor uses the target type, engagement distance, and target posture as input variables and outputs the corresponding ammunition type according to the preset military expert rule base. Ability dimension expansion stage: An ammunition selection head is serially added at the end of the policy backbone network. The ammunition selection head takes the situation feature vector output by the policy backbone network as input and uses a three-layer fully connected network. The first layer contains a linear layer, a ReLU activation function, and a Dropout layer. The second layer contains a linear layer and a ReLU activation function. The third layer is a linear output layer. Finally, the probability distribution of ammunition categories is output through the Softmax function. Progressive annealing takeover stage: The rule dependence coefficient λ(t) is controlled by an annealing scheduling function. During the annealing period, the final executed ammunition selection probability P_final is calculated, and the formula is: P_final = λ(t)·P_rule + (1 - λ(t))·P_network where s is the current battlefield state output by the high-fidelity land warfare simulation environment, target_info is the type, distance, and posture information of the current strike target, P_rule is the ammunition type probability output by the rule anchor based on the target_info, P_network is the probability distribution of ammunition categories output by the ammunition selection head based on s, and λ(t) is the rule dependence coefficient; when λ(t) decays to zero, ammunition selection is completely autonomously decided by the ammunition selection head, and t is the number of fine-tuning steps completed currently.

5. The reinforcement learning cross-simulation transfer training method for complex land warfare scenarios according to claim 4, characterized in that, The annealing scheduling function is: When t < T_warmup, λ(t) = 1.0; When t ≥ T_warmup, λ(t) = max(0, 1 - (t - T_warmup) / T_anneal); where t is the number of fine-tuning steps completed currently, T_total is the total number of fine-tuning training steps in the high-fidelity land warfare simulation environment, T_warmup is the warm-up step number and T_warmup = 0.05 × T_total, and T_anneal is the annealing period and T_anneal = 0.4 × T_total.

6. The reinforcement learning cross-simulation transfer training method for complex land warfare scenarios according to claim 1, characterized in that, The correction intensity coefficient α(k) adopts a progressive scheduling strategy: When k < T_warmup, α(k) = 0; When T_warmup ≤ k < T_warmup + T_ramp, α(k) = α_max · (k - T_warmup) / T_ramp; When k ≥ T_warmup + T_ramp, α(k) = α_max; Where T_warmup is the number of warm-up steps, T_ramp is the number of asymptotic steps, and α_max is the maximum correction intensity.

7. A reinforcement learning cross-simulation transfer training device for complex land warfare scenarios, applied to pre-constructed lightweight land warfare simulation environments and high-fidelity land warfare simulation environments, characterized in that, include: The strategy pre-training unit is used to input the mission configuration of the land warfare scenario into the lightweight land warfare simulation environment, generate simulation interaction data from the lightweight land warfare simulation environment, and use reinforcement learning algorithm to iteratively optimize and train the strategy network based on the simulation interaction data until convergence, and output the pre-trained strategy network parameters. The cross-simulation migration alignment unit is used to construct a cross-simulation state mapping module, an action space alignment module, and a reward function consistency constraint module to address the differences between the lightweight land combat simulation environment and the high-fidelity land combat simulation environment in terms of state space dimension, action space granularity, and reward value distribution. The pre-trained policy network parameters are loaded into the policy backbone network of the high-fidelity land combat simulation environment. The cross-simulation state mapping module is connected to the input end of the policy backbone network, and the action space alignment module is connected to the output end of the policy backbone network. At the same time, the reward signal is corrected and adapted based on the reward function consistency constraint module, and the transfer policy network is output. The reward function consistency constraint module employs a three-level progressive constraint mechanism, including offline bias prior modeling, pre-training reward correction, and transfer value adaptation. The offline bias prior modeling is as follows: Initial states covering various terrains and enemy / friendly situations are randomly sampled in a lightweight environment. A hybrid strategy is used to generate action sequences. The generated state-action sequences are executed in parallel in both the lightweight and high-fidelity environments. The state s_n, action a_n, and rewards returned by the two environments are recorded at each time step n. The reward difference ΔR_n = R_high^n - R_low^n is calculated, where R_high^n is the reward returned by the high-fidelity land combat simulation environment at time step n, and R_low^n is the reward returned by the lightweight land combat simulation environment at time step n. A reward difference dataset D = {(s_n, a_n, ΔR_n)} is constructed based on this dataset. A bias prediction network is trained on this dataset, and the bias prediction network outputs a predicted bias value for the reward value of a given state-action pair. The pre-training reward correction is as follows: during the training of the policy network, the deviation prediction network is used online to correct the original reward returned by the lightweight land combat simulation environment. The correction formula is R_corrected = R_raw + α(k)·ΔR_pred, where R_raw is the original reward, ΔR_pred is the deviation prediction value output by the deviation prediction network, and α(k) is the correction strength coefficient. Based on the corrected reward R_corrected, the value network is trained synchronously, and the corrected value network parameters are output. The migration value adaptation is as follows: the parameters of the corrected value network are loaded into the corrected value network V_corrected, and the high-fidelity value network V_high is randomly initialized and trained based on the real rewards returned by the high-fidelity land combat simulation environment. The temporal differential advantage estimate A_corrected of the corrected value network and the temporal differential advantage estimate A_high of the high-fidelity value network are calculated respectively. The policy is updated by progressively fusing the advantage function A_final = β(h)·A_corrected+ (1 - β(h))·A_high, where β(h) is the fusion coefficient and decays from 1 to 0 with the number of fine-tuning steps, β(h) = max(0, 1 - h / T_adapt), and T_adapt is the adaptation period. The high-fidelity environment fine-tuning unit is used to load the migration strategy network into the high-fidelity land combat simulation environment, generate high-fidelity simulation interaction data from the high-fidelity land combat simulation environment, and continuously perform reinforcement learning fine-tuning training on the migration strategy network based on the high-fidelity simulation interaction data to output a target decision strategy adapted to the high-fidelity battlefield environment.

8. A reinforcement learning cross-simulation transfer training device for complex land warfare scenarios, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program that can be executed by the processor to implement a reinforcement learning cross-simulation transfer training method for complex land warfare scenarios as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The device contains a computer program that can be executed by a processor of the device in which the computer-readable storage medium is located, to implement a reinforcement learning cross-simulation transfer training method for complex land warfare scenarios as described in any one of claims 1 to 6.

Images