An airborne radar jamming decision method based on hierarchical dimension adaptive migration

Abstract

This invention discloses an airborne radar jamming decision-making method based on hierarchical dimensional adaptive transfer. The method independently trains a hierarchical double-Q jamming decision-making model, including a meta-controller and a controller, in the source domain; reconstructs the target domain meta-controller and controller according to the actual dimensions of the target domain; establishes a semantic index mapping based on the radar state, high-level jamming strategies, and the physical meaning of parameter actions; slices and loads corresponding parameters in the input layer, output layer, and bias; maintains initialization for newly added dimensions in the target domain; and performs pruning on missing dimensions in the target domain; finally, hierarchical double-Q fine-tuning is performed in the target domain using an exponentially decaying learning rate. This invention does not require the source and target domains to have the same state or action tensor shape, and can quickly adapt to different radar models, number of operating modes, and jamming parameter sets while preserving the hierarchical tactical knowledge of the source domain, improving the initial performance, sample efficiency, convergence speed, and robustness of the target domain.

Description

Technical Field

[0001] This invention belongs to the field of electronic warfare and radar countermeasures technology, specifically relating to an airborne radar jamming decision-making method based on hierarchical dimension adaptive migration. Background Technology

[0002] Airborne radar jamming decisions based on reinforcement learning are typically trained in a simulation environment before being deployed to actual adversarial scenarios. However, different radar models vary in the number of operating modes, radio frequency range, pulse regime, and the number of configurable parameters of the jammer. Furthermore, the reconnaissance receiver introduces measurement errors in frequency, pulse width, and pulse repetition interval, causing domain shifts between the training environment and the target environment in terms of state transitions, reward distribution, and network input-output dimensions.

[0003] When the state or action dimensions of the source and target domains are inconsistent, the source domain hierarchical reinforcement learning network cannot be directly loaded according to the original tensor shape. If it is trained from scratch for each radar model and parameter combination, it not only requires a large number of target domain interaction samples, but also discards the high-level tactical knowledge already learned in the source domain, such as "prioritizing inducement in low-threat phases and prioritizing suppression in high-threat phases," as well as low-level representations such as time and frequency coverage and parameter matching. This makes it difficult to meet the requirements of electronic warfare for rapid adaptation and limited training windows.

[0004] The paper "H. You, T. Yang, Y. Zheng, J. Hao and ME Taylor, Cross-Domain Adaptive Transfer Reinforcement Learning Based on State-Action Correspondence, Proceedings of UAI 2022, PMLR 180:2299-2309, 2022" learns the correspondence between different state-action spaces through a state encoder, action encoder, and adaptive module, and transfers knowledge from multiple source policies to the target policy. However, this method requires additional training of a mapping network and cross-domain trajectory representation, and does not provide a directly implementable dimensionality slicing loading process for the input layer, output layer, and bias parameters of the radar interference hierarchical value network.

[0005] The paper "KP Sivakumar, Y. Zhang, Z. Bell, S. Nivison and MMZavlanos, Transfer Reinforcement Learning in Heterogeneous Action Spacesusing Subgoal Mapping, arXiv:2410.14484, 2024" utilizes Long Short-Term Memory (LSTM) networks to learn subgoal sequence mappings between experts and learners, aiming to improve sample efficiency in heterogeneous action spaces. However, this method relies on expert demonstrations and subgoal trajectory data, making it suitable for action semantic mapping scenarios. For radar jamming tasks where the source and target domains have the same tactical semantics but differ in network dimensions, a lightweight solution is still lacking that allows for direct reuse of shared dimensions and random initialization and synchronous fine-tuning of newly added dimensions.

[0006] As mentioned above, existing cross-domain reinforcement learning methods mostly address heterogeneous space problems through encoding, distillation, or demonstration mapping, resulting in long computational chains. Furthermore, they fail to fully utilize the structural characteristics of hierarchical interference decision-making, such as the relatively stable high-level tactical knowledge of the meta-controller and the varying input / output dimensions of the controller depending on the radar model and hardware parameters. Therefore, it is necessary to propose a dimension-adaptive transfer method for hierarchical value networks. This method replicates source domain weights by slicing according to the least common dimension without requiring paired trajectories or additional mapping networks, and uses learning rate scheduling to quickly fine-tune the target domain. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the prior art and provide an airborne radar jamming decision-making method based on hierarchical dimension adaptive transfer. The method pre-trains a hierarchical deep dual-Q network containing a meta-controller and a controller in the source domain. The target Q network is reconstructed according to the state dimension and action dimension of the target domain. Weight slices are loaded on the alignable common parts of the input layer, output layer and bias terms. The newly added dimensions in the target domain are kept randomly initialized and fine-tuned in the target domain using an exponential decay learning rate, so as to achieve rapid knowledge reuse across radar models and parameter scales.

[0008] The objective of this invention is achieved through the following technical solution: an airborne radar jamming decision-making method based on hierarchical dimension adaptive migration, the specific steps of which are as follows:

[0009] S1. Construct radar jamming decision models for the source and target domains, and determine the intersection of transferable dimensions; details are as follows:

[0010] S11. Construct radar jamming decision models for the source and target domains, and determine the state space, meta-action space, and controller action space of the source and target domains. Both the source and target domains adopt hierarchical semi-Markov radar jamming decision models. The hierarchical semi-Markov radar jamming decision models include meta-controllers and controllers. Each meta-controller and controller adopts a set of hierarchical double-Q value networks. The hierarchical double-Q value networks include online Q networks and target Q networks.

[0011] The source domain hierarchical semi-Markov radar jamming decision model is established based on the nominal number of radar modes and the nominal jamming parameter granularity; the target domain hierarchical semi-Markov radar jamming decision model redetermines the state dimension, meta-action dimension, and controller action dimension of the model based on the actual radar model, reconnaissance parameters, and jammer hardware capabilities.

[0012] S12. Calculate the minimum common dimension of each corresponding layer in the hierarchical semi-Markov radar jamming decision model of the source domain and target domain, and determine the intersection of transferable dimensions; for the meta-controller, transfer the input weights corresponding to the common radar state dimension and the output weights corresponding to the common jamming pattern; for the controller, transfer the common state and pattern input dimensions, as well as the output weights and biases corresponding to the common parameter action index.

[0013] S13. Construct initialization rules for dimensionality reduction, dimensionality increase, and parameter bias target domains; the dimensionality reduction target domain adopts a pruning loading method, retaining only the source domain weights that the target domain network can accommodate; the dimensionality increase target domain adopts common part loading, randomly initializing the newly added state columns and action rows; the parameter bias target domain is loaded as a whole when the network dimension is the same, and then the state transition and reward differences are corrected using target domain samples.

[0014] S2. Based on weight slice loading and learning rate scheduling, the target domain hierarchical perturbation strategy transfer and fine-tuning are completed: First, the meta-controller and the hierarchical depth double-Q network of the controller are trained in the source domain until convergence, and the parameters of the meta-controller and the hierarchical depth double-Q network of the controller are saved; then, the network is constructed according to the shape of the target domain and slice loading is performed; finally, the target domain state-action space, experience replay and double-Q update are used for fine-tuning.

[0015] The beneficial effects of this invention are as follows: The method of this invention directly utilizes the tensor structure of a hierarchical value network for transfer, without the need for additional training of a state-action mapping network or collection of paired expert trajectories; it replicates the source domain input layer, output layer, and bias parameters according to the least common dimension, and can simultaneously handle target domain dimensionality reduction and expansion; it retains random initialization for newly added dimensions, enabling the model to learn new radar modes and new jamming parameters; and it achieves rapid adaptation followed by stable fine-tuning through an exponentially decaying learning rate, which reduces the target domain sample requirements and mitigates catastrophic forgetting caused by a fixed large learning rate, making it suitable for cross-domain deployments with changes in radar type, operating mode, and jammer capabilities. Attached Figure Description

[0016] Figure 1 This is a flowchart of an airborne radar jamming decision-making method based on hierarchical dimension adaptive migration according to the present invention.

[0017] Figure 2 This is a comparison of the number of steps required for convergence of different training strategies in the embodiments of the present invention.

[0018] Figure 3 This is a comparison of average rewards under different training strategies in the embodiments of the present invention.

[0019] Figure 4 This is a comparison of task success rates under different training strategies in the embodiments of the present invention.

[0020] Figure 5 This is a comparison of cross-domain transfer performance under different training strategies in the embodiments of the present invention. Detailed Implementation

[0021] Layered dimensional adaptive migration refers to allowing the number of radar states, the number of controller parameter combinations, and the network input-output shapes to differ between the source and target domains while maintaining the two-layer tactical semantics of the meta-controller and the hidden layer representation structure. Deployable strategies are formed through the reuse of common tensor blocks and fine-tuning of the target domain.

[0022] This invention provides an airborne radar jamming decision-making method based on hierarchical dimension adaptive transfer. It pre-trains a hierarchical deep dual-Q network containing a meta-controller and a controller in the source domain, reconstructs the target Q network according to the state dimension and action dimension of the target domain, loads weight slices on the alignable common parts of the input layer, output layer and bias terms, keeps the newly added dimensions in the target domain randomly initialized, and fine-tunes them in the target domain using an exponentially decaying learning rate, so as to achieve rapid knowledge reuse across radar models and parameter scales.

[0023] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0024] like Figure 1 As shown, the present invention provides an airborne radar jamming decision-making method based on hierarchical dimension adaptive migration, the specific steps of which are as follows:

[0025] S1. Construct radar jamming decision models for the source and target domains, and determine the intersection of transferable dimensions; details are as follows:

[0026] S11. Construct radar jamming decision models for the source and target domains, and determine the state space, meta-action space, and controller action space for both the source and target domains. Both the source and target domains employ a hierarchical semi-Markov radar jamming decision model. This hierarchical semi-Markov radar jamming decision model includes two levels in its decision structure: a meta-controller and a controller. The meta-controller corresponds to high-level tactical decision-making, used to select high-level jamming patterns based on the radar operating mode state. The meta-controller layer inputs the one-hot encoded state vector of the radar operating mode and outputs Q-value vectors corresponding one-to-one with each high-level jamming pattern. The controller corresponds to low-level execution decision-making, used to select specific jamming parameters based on the radar state and the jamming patterns output by the meta-controller. The controller layer inputs the concatenated vector of the one-hot encoded state vector of the radar operating mode and the one-hot encoded vector of the selected high-level jamming pattern, and outputs Q-value vectors corresponding one-to-one with each specific jamming parameter combination.

[0027] The meta-controller and the controller each employ a set of hierarchical double-Q value networks (Double-HDQN), which include an online Q-network and a target Q-network. Both the online Q-network and the target Q-network use multi-layer fully connected neural networks, including an input layer, a hidden layer, and an output layer. Therefore, a complete hierarchical semi-Markov radar jamming decision model includes four Q-networks: the meta-controller online Q-network, the meta-controller target Q-network, the controller online Q-network, and the controller target Q-network.

[0028] The source domain hierarchical semi-Markov radar jamming decision model is established based on the nominal number of radar modes and the nominal jamming parameter granularity. The target domain hierarchical semi-Markov radar jamming decision model redefines the model's state dimension, meta-action dimension, and controller action dimension based on the actual radar model, reconnaissance parameters, and jammer hardware capabilities. The source domain and target domain maintain consistency in the two-stage task semantics of "induction-suppression" and the hierarchical decision semantics of "meta-controller selecting high-level jamming patterns and controller selecting specific jamming parameters." The target domain can be a reduced-dimensional environment where both state and action are smaller than the source domain, or a higher-dimensional environment where both state and action are larger than the source domain. The reduced-dimensional environment is used to represent situations with fewer radar modes or limited jammer parameter levels; the higher-dimensional environment is used to represent new equipment with richer radar modes and more executable parameter combinations.

[0029] "Source domain" and "target domain" represent two different radar countermeasure mission environments and their corresponding decision models; "corresponding layer" refers to the fully connected layer in the source domain network and the target domain network that has the same function and hierarchical semantics; "higher layer" specifically refers to the tactical decision layer where the meta-controller is located. The parameters of the source domain and target domain include: the number of radar operating modes. Number of meta-actions Number of controller actions Meta-controller input / output dimension, controller input / output dimension, hidden layer dimension In addition, there are target domain state transition and reward parameters. The parameter differences between the source domain and the target domain include: changes in the number of radar operating modes, increases or decreases in radio frequency or bandwidth levels, changes in the number of controller parameter combinations, radar parameter measurement errors, and changes in state transition probabilities.

[0030] In this embodiment, the source domain employs 8 radar modes, 3 high-level interference patterns, and 864-dimensional controller actions. The meta-controller inputs an 8-dimensional radar operating mode one-hot encoded state vector and outputs a 3-dimensional Q-value vector corresponding one-to-one with the 3 high-level interference patterns. The controller inputs an 11-dimensional concatenated vector composed of the 8-dimensional radar operating mode one-hot encoded state vector and the 3-dimensional high-level interference pattern one-hot encoded vector, and outputs the specific parameter action Q-value. Since the source domain has 8 radar operating modes, the source domain meta-controller input is an 8-dimensional radar operating mode one-hot encoded state vector. The reduced-dimensional target domain uses a 6-dimensional corresponding state vector, and the increased-dimensional target domain uses a 10-dimensional corresponding state vector.

[0031] S12. Calculate the minimum common dimension of each corresponding layer in the hierarchical semi-Markov radar jamming decision model of the source domain and the target domain, and determine the intersection of transferable dimensions.

[0032] For each online Q-network to be migrated and the target Q-network, the source domain parameter tensor name, shape, and level affiliation are read and matched with the corresponding layers in the target domain network. Only layers with consistent network semantics and hidden layer connections are included in the slice migration to avoid incorrectly copying parameters with different functions.

[0033] For the meta-controller, migrate the input weights corresponding to the shared radar state dimension and the output weights corresponding to the shared interference patterns; for the controller, migrate the shared state and pattern input dimensions, as well as the output weights and biases corresponding to the shared parameter action index.

[0034] The source domain, target domain, common dimensions, and various parameter slices are defined below:

[0035] (1) Source domain hierarchical semi-Markov radar jamming decision model; the model includes the state space of the source domain. Meta-action space Controller motion space State transition and rewards The source domain obtains a stable meta-controller through sufficient training. and controller This serves as the initialization knowledge for the target domain.

[0036] Therefore, the source domain interference decision model is constructed as follows:

[0037] (1);

[0038] in, This represents the discount factor, used to measure the importance of future returns relative to current returns, satisfying 0 ≤ <1; Let the parameter set of the source domain decision model be denoted as This includes the online controller and the target Q-network, as well as the weights and biases of each layer of the online controller and the target Q-network.

[0039] (2) Hierarchical semi-Markov radar jamming decision model for the target domain; the target domain resets its state and action space according to the new radar model or actual hardware capabilities, but maintains the hierarchical semantics of "the meta-controller selects the style and the controller selects the parameters". The state transition and reward of the target domain may have numerical deviations from those of the source domain.

[0040] Therefore, the target domain interference decision model is constructed as follows:

[0041] (2);

[0042] in, , , , and These represent the target domain's state space, meta-action space, controller action space, state transition, and reward, respectively. , or At least one of them is related to the source domain. , , Different; the set of network parameters in the target domain is denoted as This includes the online controller and the target Q-network, as well as the weights and biases of each layer of the online controller and the target Q-network.

[0043] (3) Determine the common input and output dimensions;

[0044] Suppose that in a corresponding layer of a network with the same name in the source and target domains, the input and output dimensions of the source domain are respectively... and The target domain input and output dimensions are respectively and The minimum dimension of the input from the source and target domains is used as the common input dimension, and the minimum dimension of the output is used as the common output dimension. The constructed common dimension is represented as follows:

[0045] (3);

[0046] Among them, input common dimension Determine the number of weighted columns to replicate and output the common dimensions. Determine the number of weighted rows and the number of biased elements to be copied.

[0047] (4) Slicing of the meta-controller input layer; Each input column of the meta-controller input layer corresponds one-to-one with a state component in the one-hot encoded state vector of the radar operating mode. If the number of states in the target domain decreases, redundant state columns in the source domain are pruned; if the number of states in the target domain increases, existing state columns in the source domain are copied and the random initialization of newly added state columns is retained.

[0048] Therefore, the meta-controller input weight transfer is constructed, namely:

[0049] (4);

[0050] in, This represents the minimum number of radar states in both the source and target domains. This represents the input weight matrix of the source domain meta-controller. This represents the input weight matrix of the target domain meta-controller. Representation matrix The first to the second of all rows The above formula indicates that... The first to the second of all rows Column migration to The first to the second of all rows In the case of identical hidden layer structures for column-level controllers, the weights of intermediate layers can be loaded as a whole.

[0051] (5) Controller input layer slice; the controller input is composed of radar status and element action one-hot encoded splicing. If the number of high-level interference patterns remains the same, the pattern part can be migrated as a whole; the status part only migrates the columns corresponding to the common patterns in the source and target domains.

[0052] Therefore, the controller input weight transfer is constructed as follows:

[0053] (5);

[0054] in, This represents the input layer weight matrix of the source domain controller. This represents the input weight matrix of the target domain controller; This indicates the total number of meta-actions; in this embodiment, the three interference styles have the same semantics, therefore... .

[0055] (6) Controller output layer slice;

[0056] Each row of the controller output layer corresponds to a discrete combination of disturbance parameters. When the number of actions in the target domain changes, the common action prefix is ​​determined according to the pre-defined parameter enumeration order or action semantic index, and the corresponding output line is copied.

[0057] Therefore, the controller output weight transfer is constructed as follows:

[0058] (6);

[0059] in, . This represents the output weight matrix of the source domain controller. This represents the output weight matrix of the target domain controller; Represents the first to the second of the matrix All columns of the row. If a non-prefix action mapping is used, a common action index set can be used instead of continuous slices. This embodiment uses prefix slices with a unified enumeration order.

[0060] (7) Output bias shift; the output bias directly affects the initial values ​​of each action. To maintain the source domain value ordering of shared actions, the source domain common biases are copied to the target domain, while retaining the random initialization of the biases for newly added actions.

[0061] (7);

[0062] in, This represents the output bias vector of the source domain. This represents the output bias vector of the target domain; when the number of output actions of the meta-controller is the same, its output weights and biases can be copied as a whole; when the number of actions is different, the least common output dimension is also used.

[0063] If the target domain adds an input or output dimension that does not have a directly corresponding source domain semantics, for the target domain meta-controller and the controller's online Q network, the added dimension parameters retain the random initial values ​​generated by the network initializer when the target domain is built, avoiding simple zero filling that leads to initial symmetry of neurons; after the common weight slice of the online Q network is loaded, the corresponding online Q network parameters are copied to the target Q network in whole according to Equation (13), thereby ensuring that the online Q network and the target Q network of the same decision layer have consistent initial parameters.

[0064] Therefore, we construct the initialization of the new parameters, namely:

[0065] (8);

[0066] The initialization method matches the activation function of the target domain's online Q-network, and either Kaiming Uniform or Xavier Uniform can be used for initialization.

[0067] (8) Set of transferable parameters;

[0068] All slices, full-layer hidden parameters, and corresponding biases are combined into a transferable parameter set, and the newly added dimension parameters are combined into an adaptive parameter set. Together, they constitute the initial parameter set of the online Q-network for the target domain. Then, according to equation (13), the initial parameters of the online Q-network are copied to the corresponding target Q-network.

[0069] Therefore, we construct the target parameter set, namely:

[0070] (9);

[0071] in, The parameter represents the data copied from the source domain. This indicates that the target domain is randomly initialized; both can be updated during target domain fine-tuning.

[0072] S13. Construct initialization rules for the target domain of dimensionality reduction, dimensionality increase, and parameter deviation;

[0073] The dimensionality reduction target domain adopts a pruning loading method, retaining only the source domain weights that the target domain network can accommodate; the dimensionality increase target domain adopts common part loading, randomly initializing the newly added state columns and action rows; the parameter deviation target domain can be loaded as a whole when the network dimensions are the same, and then the state transition and reward differences are corrected by using the target domain samples.

[0074] Dimension adaptation first loads the common weights and initializes the new dimensions for the online Q-networks of the target domain meta-controller and the controller, respectively. Then, it copies the adapted online Q-network parameters to the corresponding target Q-network, so that the parameters of the online Q-networks of the meta-controller and the controller are consistent with the target Q-network at the start of fine-tuning, avoiding additional deviations introduced by inconsistent initialization during the first double Q update.

[0075] In this embodiment, the source domain consists of 8 radar modes, 3 meta-actions, and 864-dimensional controller actions; the reduced-dimensional target domain consists of 6 radar modes, 3 meta-actions, and 64-dimensional controller actions; and the increased-dimensional target domain consists of 10 radar modes, 3 meta-actions, and 1350-dimensional controller actions.

[0076] The target domain network hidden layer still uses Therefore, the two hidden layers and their biases can be migrated as a whole; only the meta-controller input layer, controller input layer, and controller output layer are sliced ​​or expanded according to the target domain dimension.

[0077] S2. Based on weight slice loading and learning rate scheduling, the target domain hierarchical interference policy migration and fine-tuning are completed. Weight slice loading is performed on the source domain hierarchical double-Q network, and learning rate scheduling is used to complete the interference policy migration and fine-tuning of the target domain hierarchical double-Q network.

[0078] First, train the meta-controller and the hierarchical deep double-Q network of the controller in the source domain until convergence, and save the parameters of the meta-controller and the controller's hierarchical deep double-Q network. Then, construct the network according to the shape of the target domain and perform slice loading. Finally, fine-tune the network using the target domain state-action space, experience replay, and double-Q update, with the meta-controller and controller using separately set exponential decay learning rates. The transfer does not change the high-level tactical semantics and hierarchical reward logic of the source domain, but only changes the input-output shape of the target Q network and the target domain interaction data.

[0079] In this embodiment, in step S2, Double-HDQN is first trained in the source domain and checkpoints are saved; a network is constructed based on the target domain (dimensionality reduction or dimensionality increase); the dimensionality adaptation function is called to copy common weights layer by layer; subsequently, in the target domain, the network is trained with the initial learning rate of the meta-controller. Initial learning rate of the controller Make fine adjustments and decay to .

[0080] The migration and fine-tuning of the target domain hierarchical interference strategy includes the following sub-steps:

[0081] S21. Source Domain Pre-training: The meta-controller and controller's hierarchical deep double-Q networks are pre-trained in the source domain, and network parameters are extracted. A periodic re-evaluation strategy, internal and external hierarchical rewards, and double-Q updates using online Q network selection / target Q network evaluation are employed. The meta-controller and controller are trained in a nominal environment, preserving the weights, biases, and target Q network parameters of each fully connected layer. During the source domain training phase, both the meta-controller and controller employ an ε-greedy strategy for action selection: randomly selecting actions from the current set of available actions with probability ε, and selecting the action with the largest Q-value output by the online Q network with probability 1-ε. ε is gradually decreased as training progresses to transition from full exploration to stable utilization. The meta-controller maintains the high-level interference pattern within a fixed tactical period, while the controller is gradually configured with specific interference parameters. The online Q network selects actions, and the target Q network evaluates actions until the two-stage task rewards and success rates in the source domain stabilize.

[0082] The source domain training can be performed using 4000 rounds and 20 random seeds for performance verification. During deployment and migration, only convergence checkpoints are required, and it is not necessary to revisit all source domain interaction trajectories.

[0083] S22. Target Domain Network Construction: Construct new meta-controllers and controllers based on the target domain dimension; both meta-controllers and controllers adopt hierarchical deep double-Q networks, and the hidden layer structure in the network is consistent with the source domain network of the same name. The input layer and output layer are reset according to the number of target domain states, the number of meta actions, and the number of controller actions.

[0084] Traverse the same-named parameters of the online Q network of the source and target domains, record the shape of the source parameter, the shape of the target parameter, the range of common slices, and whether the entire layer can be copied, and form an adaptation list;

[0085] The input-output mapping of the target domain online Q-network can be represented as:

[0086] (10);

[0087] in, Indicates the target domain meta controller. Indicates the target domain controller; This represents the dimension of the one-hot encoded state vector of the target domain radar operating mode. Indicates the number of high-level interference styles in the target domain. This indicates the number of specific disturbance parameter actions of the target domain controller. Therefore, the meta-controller input dimension is... Output dimension is ; Represents the real number domain; the controller input is composed of one-hot encoded data of the target domain radar state and the upper-level interference pattern, and its input dimension is... + The output dimension is .

[0088] S23, Parameter Slicing: Load the meta-controller and controller parameters according to the adapter list; perform full tensor copying on hidden layers with identical shapes, perform common slice copying on the input and output layers, and do not cover the excess parts of the target domain; after loading, obtain the initial parameters of the target domain meta-controller. and controller initial parameters , is represented as:

[0089] (11);

[0090] The function processes common weights and new parameters according to equations (3) to (8).

[0091] (12);

[0092] (13);

[0093] In equation (13), the superscript Let k represent the target Q-network, where k is either meta or ctrl, corresponding to the meta controller and the controller, respectively. Equation (13) is used to initialize the target Q-network: After the online Q-network slices of the target domain are loaded, the parameters of the online Q-network are copied to the corresponding target Q-network to ensure that the parameters of the online Q-network and the target Q-network of the same decision layer are consistent at the beginning of dual Q learning.

[0094] Repeat steps S22-S23 to complete the parameter adaptation of the four sets of the meta-controller online / target Q network and the controller online / target Q network.

[0095] S24. Layered Fine-Tuning: In the target domain, ε-greedy exploration is adopted, and layered fine-tuning is performed according to an exponentially decaying learning rate. During the target domain fine-tuning phase, the meta-controller and the controller continue to select actions using an ε-greedy strategy, with ε gradually decreasing as training progresses. At the same time, the target domain training progress is normalized, and the meta-controller and the controller use different initial learning rates but the same minimum learning rate and decay rate, so that high-level tactical knowledge adjusts relatively slowly, while low-level parameter knowledge adapts to the target domain more quickly.

[0096] Secondly, fine-tune the learning rate scheduling of the adapted network in the target domain:

[0097] (14);

[0098] (15);

[0099] (16a);

[0100] (16b);

[0101] in, Indicates the normalized training progress. Indicates the attenuation rate; in equations (14) and (15), This represents the training progress of the normalized target domain, and its value ranges from [0,1]. ( )and ( The numbers ) represent the training progress respectively. The time-based controller and the controller's learning rate; and These represent the initial learning rates of the meta-controller and the controller, respectively. This represents the minimum learning rate that can be allowed to decay. This represents the exponential decay coefficient. Equation (16a) is the general formula for gradient update of the online Q-network in the target domain, where... Pick or These correspond to the meta-controller and the controller, respectively. Indicates the target domain. Online Q-network parameters for each decision-making layer This indicates the current learning rate of the decision-making team. This represents the loss function of the online Q-network in the target domain. The loss function is expressed as follows: The gradient. Equation (16b) represents the periodic synchronization method of the target Q-network, where Indicates the target domain. The target Q-network parameters of each decision layer `C` represents the synchronization interval of the target Q-network, and `episode` represents the current training epoch of the target domain. When the `episode` meets the preset synchronization conditions, the parameters of the online Q-network are copied to the corresponding target Q-network. That is, every `C` training epochs, the parameters of the online Q-network in the target domain are copied to the corresponding target Q-network. The online Q-network is continuously updated through gradient descent, while the target Q-network does not participate in each step of gradient descent, but only copies parameters from the online Q-network at fixed intervals, thus maintaining the relative stability of the Double-Q target value. A larger learning rate can be used initially, which is beneficial for the rapid formation of effective new action dimensions. In the later stages, a smaller learning rate can be used to reduce drastic disturbances to the transferred public weights and minimize the destruction of effective knowledge in the source domain.

[0102] Secondly, a new experience replay pool is established in the target domain, and the two-layer network is updated based on the target domain rewards:

[0103] (17);

[0104] (18);

[0105] (19);

[0106] In equation (17), This represents a sample of interaction experience in the target domain. This indicates the current state of the target domain meta controller. Indicates the current state of the target domain controller. This indicates the high-level interference pattern selected by the target domain meta controller. This indicates the specific interference parameter action selected by the target domain controller. This represents the intrinsic reward obtained by the target domain controller. This represents the extrinsic reward obtained by the target domain meta controller. This indicates the next state of the target domain meta controller after the action is performed. This represents any action within the action space of the target domain controller. This represents any action in the action space of the target domain meta-controller. In equation (18), This indicates that the target domain controller updates the target value using the dual-Q algorithm. and These represent the target domain controller online Q network and the target Q network, respectively. Indicates the next state of the target domain controller after the action is performed; This represents the discount factor. In equation (19), This indicates that the target domain controller updates the target value using the dual-Q algorithm. This represents the cumulative extrinsic reward over the duration of a high-level interference pattern, which is composed of multiple... ; This indicates the number of environmental time steps that represent the duration of the high-level interference pattern. This embodies the semi-Markov time span discount. Specifically, target domain experience is not mixed with source domain replay data to prevent the old transition distribution from dominating fine-tuning in the long term; source domain knowledge enters the target domain only through the initial network weights. Target domain fine-tuning still maintains the hierarchical reward and periodic policy re-evaluation logic of the source domain, but all states, actions, and transitions originate from the target domain, ensuring that common knowledge and the newly added dimensions in the target domain converge together.

[0107] The learning rate of the meta controller is determined by equation (14), the learning rate of the controller is determined by equation (15), the target Q network synchronization is determined by equation (16), and the two-layer dual Q target is determined by equations (18) and (19).

[0108] S25. Use the fine-tuned network to generate a deterministic strategy for the target domain. After the target domain fine-tuning is complete, set ε to 0 and disable random exploration; disable After random exploration, the meta-controller selects based on the target domain state. The controller selects the most significant interference pattern from the target domain action set. The maximum interference parameters are continuously reassessed periodically based on changes in radar mode:

[0109] (20)

[0110] (twenty one)

[0111] (twenty two)

[0112] In equation (20), This indicates the high-level interference pattern ultimately selected by the target domain meta controller. This represents the online Q-network of the target domain meta-controller; in equation (21), This indicates the specific disturbance parameter action ultimately selected by the target domain controller. Let represent the target domain controller's online Q-network, where the controller selects specific parameter actions under the constraints of the high-level disturbance pattern determined by equation (20). In equation (22), Gain represents the final average reward gain rate of the transfer fine-tuning relative to the target domain trained from zero. This represents the final average reward of the migration fine-tuning strategy in the target domain. This represents the final average reward of a strategy trained from scratch on the same target domain.

[0113] The target domain deterministic strategy includes both high-level jamming pattern decisions and low-level jamming parameter decisions. Periodic strategy reassessment recalculates the Q-values ​​of all high-level jamming patterns under the current radar state according to a fixed tactical time window, and reselects the high-value high-level strategy. The processing method is as follows:

[0114] (1) During a tactical cycle, the meta controller maintains the current high-level interference pattern and the controller continuously selects specific parameters;

[0115] (2) When the tactical cycle expires, read the latest radar operating mode status and recalculate the Q value of each high-altitude interference pattern;

[0116] (3) If the original high-level interference pattern still has the maximum Q value, then the original strategy will continue to be implemented;

[0117] (4) If another higher-level interference pattern has a larger Q value, then switch to the new higher-level interference pattern;

[0118] (5) If the change in radar operating mode indicates that the induction phase has been completed, the high-level strategy will shift to a strategy that is more favorable to the suppression phase during the next reassessment.

[0119] (6) If the complete “inducement-suppression” task has been completed, the current round will be terminated and no further evaluation will be conducted in the next cycle.

[0120] S26. Output the adapted target domain element controller, target domain controller, and the cross-domain airborne radar jamming strategy composed of the two.

[0121] The migration architecture consists of three main parts: the first part is a source domain hierarchical deep double-Q network, which is used to form stable high-level tactical and low-level parameter knowledge; the second part is a dimension adaptation module, which is used to compare the tensor shapes of each network layer in the source domain and the target domain and copy common weight blocks; the third part is a target domain fine-tuning module, which is used to update the migrated parameters and new parameters using target domain interaction samples.

[0122] In this embodiment, the airborne radar jamming decision-making method further includes step S3, which involves performing a migration performance test, as follows:

[0123] The source domain is set with 8 radar modes and 864-dimensional controller actions; the reduced-dimensional target domain has 6 radar modes and 64-dimensional controller actions; and the increased-dimensional target domain has 10 radar modes and 1350-dimensional controller actions. Both types of target domains maintain the two-stage "induction-suppression" mission.

[0124] We compared training from zero, fixed learning rate transfer, and exponentially decaying learning rate transfer. The target domain was trained for 400 to 800 rounds, with a maximum of 500 steps per round, using 10 random seeds. The evaluation metrics were average reward, cumulative interaction steps, task success rate, and final performance gain.

[0125] In this embodiment, the source domain pre-trained network is loaded into the target Q network after slice adaptation. The common state and action dimensions inherit the weights of the source domain, and the newly added dimensions are kept randomly initialized. The universality of the present invention for the two dimensionality change directions is verified by comparing the learning curves of the dimensionality reduction and dimensionality increase scenarios.

[0126] Based on the network dimensions, slice range, and experimental results of the source and target domains, the results are shown in Table 1.

[0127] Table 1

[0128]

[0129] like Figure 2 As shown, the cumulative environmental interaction steps for zero-based training, fixed-learning-rate transfer, and exponentially decaying learning-rate transfer are 23,846, 22,051, and 22,123 steps, respectively. The cumulative steps for both fixed-learning-rate transfer and exponentially decaying learning-rate transfer are lower than those for zero-based training, indicating that by loading weights from the source domain network that match the target domain dimension, the target domain agent can reduce ineffective exploration and more quickly form parameter selection rules capable of completing the two-stage interference task. The similar cumulative steps for fixed-learning-rate transfer and exponentially decaying learning-rate transfer indicate that both transfer methods have a significant effect on reducing environmental interaction costs. The cumulative step count mainly reflects the improvement in sample efficiency brought about by transfer initialization, while the long-term performance difference between the two learning rate strategies needs further evaluation in conjunction with average reward and task success rate.

[0130] like Figure 3 As shown, the average rewards of the three training strategies all increase rapidly with the increase of training rounds, and eventually stabilize in the high-reward range of approximately 35 to 36, indicating that all three methods can learn effective perturbation strategies in the target domain. Compared with training from zero, fixed learning rate transfer and exponential decay learning rate transfer can more smoothly enter the high-reward range in the early stage of training, indicating that the high-level tactical organization knowledge and low-level parameter matching knowledge in the source domain pre-trained network can continue to play a role in the target domain. Figure 3 The exponentially decaying learning rate transfer reached a high reward level in the mid-to-late stages and showed good stability, indicating that dynamic learning rate scheduling is beneficial for coordinating the preservation of source domain knowledge and the adaptation of target domain characteristics.

[0131] like Figure 4As shown, the task success rates of all three training strategies gradually increase with the number of training rounds. Training from scratch, lacking pre-trained knowledge from the source domain, requires relearning the stage organization logic of the meta-controller and the parameter configuration rules of the controller in the target domain. Its success rate improvement is relatively slow, stabilizing at around 0.65 in the later stages of training. Fixed learning rate transfer and exponential decay learning rate transfer both load common parameters from the source domain network through dimensionality adaptation; therefore, the overall task success rate of these two transfer methods is higher than that of training from scratch. This result indicates that the "inducing first, then suppressing" high-level strategy organization rules learned by the source domain meta-controller, as well as the partial value representation formed by the controller in terms of time-frequency coverage and interference parameter configuration, can be effectively transferred to the target domain and improve the probability of completing the complete two-stage task. Exponential decay learning rate transfer can quickly adapt to the target domain dynamics in the early stages of fine-tuning and protect the transferred effective weights in the later stages by reducing the learning rate, thus exhibiting better overall performance in terms of task success rate and training stability. Figure 4 Further evidence shows that the performance improvement of this invention is not merely an increase in single-step reward, but rather an increase in the actual completion probability of the complete "inducement-suppression" task, indicating that dimensional adaptation and migration fine-tuning can preserve the hierarchical tactical structure of the source domain in the target domain.

[0132] like Figure 5 As shown, Figure 5 (a) represents the average reward change in the dimensionality reduction target domain. Figure 5 (b) shows the average reward change in the dimensionality-up target domain. In the dimensionality-down target domain, transfer fine-tuning achieves a higher average reward early in training and quickly enters a stable reward range, indicating that when the state and action dimensions of the target domain are smaller than those of the source domain, the common input columns, common output rows, and bias parameters corresponding to the target domain in the source domain network can directly provide effective initialization. In the dimensionality-up target domain, the advantages of transfer fine-tuning over training from scratch are more pronounced. Transfer fine-tuning achieves positive rewards from the start of training and quickly improves to a higher reward range; training from scratch has a negative or low reward phase in the early stages of training, requiring more training rounds to form an effective policy. In the later stages of training, the average reward of transfer fine-tuning remains higher than that of training from scratch. Figure 5 The results show that the dimension adaptation method of the present invention can be used for both dimension reduction migration where the target domain dimension is smaller than the source domain and dimension increase migration where the target domain dimension is larger than the source domain, and has more obvious advantages in sample efficiency and final performance in the dimension increase target domain.

[0133] comprehensive Figures 2 to 5It is known that weight slicing can reduce the number of environmental interaction steps required for target domain training and improve the initial value representation of the target domain network; exponential decay learning rate can maintain strong adaptability in the early stage of transfer learning and reduce perturbation of common weights in the later stage of training; transfer training can improve the success rate of the complete two-stage task and is applicable to both dimensionality-reduced and dimensionality-increasing target domains. Therefore, this invention can achieve effective reuse of hierarchical perturbation decision knowledge when the state dimension or action dimension of the source domain and the target domain are inconsistent, thereby improving the sample efficiency, convergence speed, and final decision performance of target domain training.

[0134] In summary, to address the issue of inconsistent state and action dimensions when deploying airborne radar jamming decision models from the simulated source domain to the actual target domain, this invention first determines the hierarchical network shape and common dimensions of the source and target domains; then, it performs slicing or full-layer loading on the meta-controller and the input, output, bias, and hidden layers of the controller; secondly, it maintains random initialization for newly added dimensions and performs double-Q fine-tuning in the target domain using an exponentially decaying learning rate; finally, it outputs a target domain strategy that can adapt to new radar modes and new jamming parameter scales, achieving efficient and stable cross-domain knowledge reuse without relying on paired trajectories and additional mapping networks.

[0135] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. An airborne radar jamming decision-making method based on hierarchical dimensional adaptive migration, characterized in that, The specific steps are as follows: S1. Construct radar jamming decision models for the source and target domains, and determine the intersection of transferable dimensions; details are as follows: S11. Construct radar jamming decision models for the source and target domains, and determine the state space, meta-action space, and controller action space for the source and target domains; both the source and target domains adopt hierarchical semi-Markov radar jamming decision models. The hierarchical semi-Markov radar jamming decision model includes a meta-controller and a controller. Each meta-controller and controller adopts a set of hierarchical double-Q value networks. The hierarchical double-Q value networks include an online Q network and a target Q network. The source domain hierarchical semi-Markov radar jamming decision model is established based on the nominal number of radar modes and the nominal jamming parameter granularity; the target domain hierarchical semi-Markov radar jamming decision model redetermines the state dimension, meta-action dimension, and controller action dimension of the model based on the actual radar model, reconnaissance parameters, and jammer hardware capabilities. S12. Calculate the minimum common dimension of each corresponding layer in the hierarchical semi-Markov radar jamming decision model of the source domain and target domain, and determine the intersection of transferable dimensions; for the meta-controller, transfer the input weights corresponding to the common radar state dimension and the output weights corresponding to the common jamming pattern. For the controller, migrate the common state and style input dimensions, as well as the output weights and biases corresponding to the common parameter action index; S13. Construct initialization rules for dimensionality reduction, dimensionality increase, and parameter bias target domains; the dimensionality reduction target domain adopts a pruning loading method, retaining only the source domain weights that the target domain network can accommodate; the dimensionality increase target domain adopts common part loading, randomly initializing the newly added state columns and action rows; the parameter bias target domain is loaded as a whole when the network dimension is the same, and then the state transition and reward differences are corrected using target domain samples. S2. Based on weight slice loading and learning rate scheduling, the target domain hierarchical perturbation strategy transfer and fine-tuning are completed: First, the meta-controller and the hierarchical depth double-Q network of the controller are trained in the source domain until convergence, and the parameters of the meta-controller and the hierarchical depth double-Q network of the controller are saved; then, the network is constructed according to the shape of the target domain and slice loading is performed; finally, the target domain state-action space, experience replay and double-Q update are used for fine-tuning.

2. The airborne radar jamming decision-making method based on hierarchical dimension adaptive migration according to claim 1, characterized in that, Step S2 includes the following sub-steps: S21. Source Domain Pre-training: Pre-train the hierarchical deep double-Q network of the meta-controller and the controller in the source domain, and extract the network parameters; both the meta-controller and the controller adopt the ε-greedy strategy to select actions; S22. Target Domain Network Construction: Construct new meta-controllers and controllers based on the target domain dimension; both meta-controllers and controllers adopt hierarchical deep double-Q networks, and the hidden layer structure in the network is consistent with the source domain network of the same name. The input layer and output layer are reset according to the number of target domain states, the number of meta-actions, and the number of controller actions. Traverse the same-named parameters of the online Q network of the source and target domains, record the shape of the source parameter, the shape of the target parameter, the range of common slices, and whether the entire layer can be copied, and form an adaptation list; S23, Parameter Slicing: Load the meta-controller and controller parameters according to the adapter list; perform full tensor copying on hidden layers with identical shapes; perform common slice copying on input and output layers; do not cover the extra parts of the target domain; after loading, obtain the initial parameters of the target domain meta-controller and the initial parameters of the controller respectively. Repeat steps S22-S23 to complete the four sets of parameter adaptations for the online / target Q network of the meta controller and the online / target Q network of the controller; S24, Layered Fine-tuning: In the target domain, ε-greedy exploration is adopted and layered fine-tuning is performed according to the exponentially decaying learning rate; in the target domain fine-tuning stage, the meta-controller and controller continue to select actions using ε-greedy strategies respectively; secondly, a new experience replay pool is established in the target domain and the two-layer network is updated according to the target domain reward. S25. Use the fine-tuned network to generate a deterministic strategy for the target domain. After the target domain fine-tuning is complete, set ε to 0 and disable random exploration; disable After random exploration, the meta-controller selects based on the target domain state. The controller selects the most significant interference pattern from the target domain action set. The largest interference parameter; S26. Output the adapted target domain element controller, target domain controller, and the cross-domain airborne radar jamming strategy composed of the two.

Images