Joint resource scheduling and transmission zeroing method and system based on reinforcement learning

Abstract

The invention belongs to the field of radar signal processing, and discloses a joint resource scheduling and transmission zeroing method and system based on reinforcement learning. According to the method, in a scene that a plurality of transmitting nodes and a plurality of receiving nodes cooperatively execute a target tracking task in a receiving and transmitting separated MIMO radar network, aiming at the problem that a traditional radar resource allocation method lacks dynamic real-time adjustment capability and lacks comprehensive consideration on long-term performance of radar system resources, firstly, the dynamic real-time adjustment capability and the comprehensive consideration on the long-term performance of the radar system resources are considered; according to the method, joint optimization is carried out by considering transmission zero setting and resource scheduling, emission waveforms, residence time and power distribution of nodes are taken as optimization variables, then, cumulative signal-to-noise ratio constraints and the low interception probability of a whole task are considered at the same time, an optimization model for minimizing the interception probability is provided, and finally, for the optimization model, the optimization model is optimized. An algorithm based on reinforcement learning is adopted for solving, specifically, a PPO algorithm is adopted, and finally the low interception probability and the resource management efficiency are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of radar signal processing, and in particular relates to a joint resource scheduling and transmission zeroing method and system based on reinforcement learning. Background Art

[0002] With the evolution of modern warfare and the improvement of information warfare requirements, radar systems are key equipment for obtaining battlefield situation awareness, and their performance is directly related to the success or failure of operations. In particular, in complex electronic countermeasure environments, traditional radar systems face severe challenges. These challenges mainly come from the enemy's advanced jamming technology, stealth technology and multi-target tracking requirements, which require radar systems to have higher detection accuracy, stronger anti-jamming capabilities and better resource management strategies. Traditional radar systems usually use fixed waveforms, power and scanning modes. This static resource configuration method is inadequate in a changing battlefield environment. Especially when fighting against stealth targets and electronic jamming, traditional methods often fail to effectively reduce the probability of being detected and interfered by the enemy, while ensuring high-precision detection of targets. In addition, with the diversification of combat missions, radar systems need to achieve fast and accurate tracking of multiple targets while maintaining a low probability of intercept (LPI), which puts higher requirements on the dynamic scheduling of radar resources. In recent years, radar resource management technology based on artificial intelligence has received widespread attention. Among them, reinforcement learning, as an effective machine learning method, continuously optimizes strategies through interactive learning with the environment to achieve decision optimization in complex environments. However, most of the existing research on radar resource scheduling based on reinforcement learning focuses on single resource optimization, such as waveform design or power allocation, without considering the application in low intercept radar. In particular, there is a lack of comprehensive scheduling strategies for radar system resources, especially in the joint consideration of transmission zeroing and resource scheduling. Transmission zeroing technology is one of the important means to improve the LPI performance of radar. It creates zero points in radar beamforming to reduce the radiation intensity in a specific direction, thereby reducing the probability of being intercepted by the enemy detection system. However, traditional methods often require a lot of computing resources and time, and it is difficult to meet real-time requirements. In addition, these methods usually only optimize according to the current environment, without considering the impact of radar resource scheduling on long-term performance, resulting in limited improvement in the overall performance of the mission.

[0003] Through the above analysis, the problems and defects of the prior art are as follows:

[0004] Traditional radar resource allocation methods lack dynamic real-time adjustment capabilities and are difficult to adapt to complex and changing electronic warfare environments. Existing radar resource scheduling research mostly focuses on single resource optimization and lacks comprehensive consideration of the long-term performance of radar system resources. Summary of the invention

[0005] The purpose of the present invention is to provide a joint resource scheduling and transmission zeroing method based on reinforcement learning, which can achieve low interception probability while satisfying the signal-to-noise ratio (SNR) constraint, and optimize the transmission waveform, dwell time and power allocation to improve the low interception probability and resource management efficiency.

[0006] The present invention is implemented as follows: a joint resource scheduling and transmission zeroing method based on reinforcement learning, the joint resource scheduling and transmission zeroing method based on reinforcement learning comprises:

[0007] Step 1: Establish a system model, considering a MIMO radar network with separate transmission and reception, including multiple transmitting nodes and multiple receiving nodes to perform tasks collaboratively;

[0008] Step 2: Model the problem, establish a resource allocation model, define relevant variables such as transmission signal, power, signal-to-noise ratio, low interception probability, etc., and propose a target optimization function on this basis to minimize the interception probability of the entire task while satisfying the task cumulative signal-to-noise ratio and real-time signal-to-noise ratio;

[0009] Step 3: Solve the resource allocation model and develop a deep reinforcement learning-based strategy for real-time scheduling and transmission zeroing of radar resources. This strategy optimizes radar resource usage through interactive learning with the environment, including selecting transmit waveforms, adjusting dwell time, and dynamically allocating power. At the same time, the proximal policy optimization (PPO) algorithm is used to solve the resource allocation model to achieve fast transmission zeroing, reduce the probability of radar signals being intercepted, and improve the stealth performance of the radar system and its effective detection and tracking capabilities in complex electronic warfare environments.

[0010] Furthermore, in step 1, consider a MIMO radar network in which both the transmitting node and the receiving node are Q, and each node is a MIMO radar cooperatively tracking the target.

[0011] Further, in step 2, the problem is modeled.

[0012] The matrix of the transmitted signal at time t can be expressed as:

[0013]

[0014] x t,m,q =[x t,m,q (1),x t,m,q (2),…,x t,m,q (L)] is a row vector of size L, representing the waveform transmitted by the qth antenna, with a sampling number of L. The composite signal in the θ direction can be expressed as:

[0015]

[0016] At,m is the amplitude coefficient of the waveform matrix transmitted by radar m at time t. θ is the emission direction vector, which can be expressed as:

[0017]

[0018] Where d is the distance between the transmitting antennas and λ is the signal wavelength. The average power in the θ direction during the pulse duration can be expressed as:

[0019]

[0020] In order to achieve LPI while ensuring detection performance, the signal-to-noise ratio and intercept probability are considered. At time t, the signal-to-noise ratio of radar m to target n can be expressed as:

[0021]

[0022] Where N t,m is the cumulative number of correlations, P t,m,n G is the transmission power of radar m to target n at time t. recive is the gain of the radar receiving antenna. Noise is the noise power of the radar receiving system, which can be obtained by noise = KT r B r F r Calculate. K is the Boltzmann constant. Tr is the absolute temperature of the radar receiver. Br is the bandwidth of the radar receiver, and Fr is the noise factor of the radar receiver. β t,m,n is the attenuation, taking into account the target scattering and propagation distance, which can be expressed as:

[0023]

[0024] In the formula, σ n is the radar cross section (RCS) of the detected target n, R t,m,n is the distance between radar m and target n at time t.

[0025] For LPI performance, considering the interception probability of radar m at time t, it can be expressed as:

[0026]

[0027] Where T t,m =N t,m T pulse , residence time T t,m It can be expressed as N t,m T pulse is the pulse duration of the transmitted waveform, T j is the search time of the interceptor. The error complement function (·) is the complementary error function. p′ fais the interceptor false alarm probability. t,m,j G is the transmission power of radar m in the direction of interceptor j. r is the gain of the interceptor receiving antenna. G P is the processing gain of the interceptor. t,m,j is the distance between radar m and interceptor j. 0 is the absolute temperature of the interception receiver. B is the bandwidth of the interception receiver, and F is the noise factor of the interception receiver. From equations (5) and (7), we can see that SNR t,m,n and p t,m N t,m ,A t,m and X t,m In our method, the emission waveform matrix can be expressed as a function:

[0028] X t,m =opt(a t,m ) (8)

[0029] Where a t,m It is a parameter that controls the proportion of transmission power in each target direction. It can realize power control in different target directions and real-time transmission zero value in the interceptor direction. t,m ,A t,m and a t,m , resource management can be used to achieve performance optimization.

[0030] It should be noted that the above considerations are radars. For networked radars, the signal-to-noise ratio of target n at time t can be described as:

[0031]

[0032] The probability that at least one radar is intercepted at time t can be described as:

[0033]

[0034] While satisfying the mission cumulative signal-to-noise ratio and real-time signal-to-noise ratio, the interception probability of the entire mission is minimized, which can be expressed as:

[0035]

[0036] Where T is the time step of the entire task.

[0037] Furthermore, in step three, a learning strategy based on reinforcement learning is constructed, which optimizes the strategy through the interaction between the agent and the environment. This process is guided by rewards that indicate the desirability of the agent's behavior. In this study, we apply proximal policy optimization (PPO) to resource scheduling of network radar systems.

[0038] The states, observations, actions, and rewards are presented separately. The state of the mth agent is defined as:

[0039]

[0040] T is the time step. and are the two-dimensional coordinates of the target n.

[0041] is the cumulative signal-to-noise ratio of target n from time step 0 to t-1. and are the 2D coordinates of the interceptor. The state contains complete information describing the current environment, including the positions of the target and interceptor, the accumulated signal-to-noise ratio, and the time step.

[0042] The observation value of the mth agent is defined as:

[0043]

[0044] and are the angle and distance from radar m to target n at time step t, respectively. and is the angle and distance from radar m to interceptor j, t is the time step, is the cumulative signal-to-noise ratio of target n from time step 0 to t-1. The observation result is the information that the radar can obtain, and the observation result is input into the decision network to determine the action at the next moment.

[0045] The action of the mth agent is defined as:

[0046]

[0047] The action is the control of the mth radar, which describes the emission of radar m. It consists of the accumulation number N associated with the dwell time t,m , and the amplitude coefficient A associated with the nulled beam pattern t,m and the scale parameter a t,m .

[0048] The reward of the mth agent is defined as:

[0049]

[0050] Among them, r m,1 t , r m,2 t , r m,3 t are the reward of interception probability in the loss function, the constraint of cumulative signal-to-noise ratio, and the constraint of real-time signal-to-noise ratio. Specifically, rm,1 t Designed for:

[0051]

[0052] The value range is -0.5 to 0.5. 1 is the interception probability reward coefficient. To implement the constraints in equation (11), the cumulative signal-to-noise ratio r is introduced m,2 t The constraint reward is designed as follows:

[0053]

[0054] It is the overall constraint for the entire task. At the end of the task, the cumulative signal-to-noise ratio is calculated. m,2 t To achieve real-time signal-to-noise ratio constraints, the design is similar to:

[0055]

[0056] The detailed execution steps of the algorithm are as follows:

[0057] 1. Initialize the policy network π and set the parameters ψ, initialize the value network V and set the parameters φ, and initialize the hyperparameters of PPO.

[0058] 2. For each training number n train From 1 to N train Perform the following steps:

[0059] 3. Initialize the playback buffer D of the mth radar m . Initialize the target and interceptor positions.

[0060] 4. For each radar m from 1 to M, perform the following steps:

[0061] 5. Get the status of the mth radar and noisy observations

[0062] 6. Calculate Actions

[0063] 7. Calculate X t,m According to equation (7), the maximum value p in the noise range is calculated t,m , calculate p according to equation (10) t According to equation (5), the minimum signal-to-noise ratio SNR in the noise range is calculated t,m,n , calculate SNR according to equation (9) t,n . Calculate the reward r according to equation (15) t .

[0064] 8. If t < T, perform the following steps:

[0065] 9. The target and interceptor fly to the next position randomly and calculate and

[0066] 10. Otherwise:

[0067] 11. Initialize the positions of the target and interceptor.

[0068] 12. Calculate the advantage function

[0069] 13. For each radar m from 1 to M, perform the following steps:

[0070] 14. Stored in playback buffer D m middle.

[0071] 15. For each update number n update From 1 to N update Perform the following steps:

[0072] 16. For each radar m from 1 to M, perform the following steps:

[0073] 17. From D m Sample a batch of transformations in . Update the parameters of the policy network and the value network according to PPO.

[0074] 18. The algorithm ends.

[0075] The policy network π is used to control the action of the radar, and the value network V is used to estimate the cumulative benefits, which is the calculation advantage function A t,m Each radar interacts with the environment according to the policy network π, collecting data from step 2 to step 17. The policy network π and the value network V are updated from step 13 to step 18.

[0076] Another object of the present invention is to provide a radar system for joint resource scheduling and transmission zeroing based on reinforcement learning, and the radar system for joint resource scheduling and transmission zeroing based on reinforcement learning includes:

[0077] The system model building module is used to build a networked radar system consisting of a MIMO radar with separate transmitters and receivers. Multiple transmitting nodes and multiple receiving nodes of the radar system cooperate to perform target detection and tracking tasks;

[0078] The resource optimization model building module is used to minimize the interception probability and build an optimization model for joint resource scheduling and transmission zeroing under the conditions of satisfying the real-time signal-to-noise ratio constraints and the cumulative signal-to-noise ratio constraints, taking the radar's transmitting waveform, dwell time and power allocation as optimization variables.

[0079] The resource optimization model solving module adopts an algorithm based on reinforcement learning and applies the PPO algorithm to realize real-time intelligent scheduling of radar resources, and realizes transmission zeroing in a data-driven manner.

[0080] Another object of the present invention is to provide a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the joint resource scheduling and transmission zeroing method based on reinforcement learning.

[0081] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to perform the steps of the joint resource scheduling and transmission zeroing method based on reinforcement learning.

[0082] Another object of the present invention is to provide an information data processing terminal, which is used to implement the radar system with joint resource scheduling and transmission zeroing based on reinforcement learning.

[0083] In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:

[0084] First, the present invention can significantly reduce the probability of radar being intercepted while satisfying the signal-to-noise ratio (SNR) constraint by jointly optimizing the radar transmission waveform, power, dwell time and other working parameters when the radar system performs the target tracking task. Compared with the traditional optimization method, the resource scheduling method based on reinforcement learning adopted by the present invention can take into account the cumulative SNR and interception probability of the entire task in real time, and realize radar resource management in a dynamic environment.

[0085] The present invention provides a joint resource scheduling and transmission zeroing method based on reinforcement learning. Under the conditions of satisfying real-time signal-to-noise ratio constraints and cumulative signal-to-noise ratio constraints, the radar's transmission waveform, power and dwell time are used as optimization variables to minimize the radar interception probability, establish a resource allocation model, and use a reinforcement learning-based method to solve it.

[0086] Second, the advantages and positive technical effects of the present invention are also reflected in:

[0087] Optimize computational efficiency: The deep reinforcement learning method used in the present invention greatly reduces the difficulty and computational complexity of solving the optimization problem by converting the complex mixed integer nonlinear optimization problem into a data-driven decision problem. Compared with traditional optimization algorithms, the method of the present invention can provide solutions faster and meet real-time requirements.

[0088] Adaptive resource management: The method of the present invention can dynamically adjust the resource allocation strategy based on real-time feedback. When the system detects that the tracking error of a target increases, the algorithm can automatically increase the resource investment in the target (such as increasing the transmission power or adjusting the beam pointing) to improve the tracking performance.

[0089] Real-time transmission nulling: The present invention achieves fast transmission nulling in the direction of the interceptor by adjusting the radar beamforming in real time, while ensuring high gain in the direction of the target. This real-time nulling technology effectively reduces the risk of the radar system being detected by the enemy and enhances the survivability of the radar.

[0090] Overall performance improvement: The present invention can effectively reduce the probability of being intercepted during the entire mission while ensuring tracking accuracy. Through simulation experiments, it is verified that the method of the present invention has significantly improved low interception performance compared with traditional methods such as particle swarm optimization (PSO).

[0091] Flexibility and robustness: The method of the present invention is not only applicable to single target tracking, but can also be extended to multi-target tracking scenarios. By flexibly adjusting the strategy network, the present invention can adapt to different task requirements and environmental changes, showing good robustness.

[0092] Third, the technical solution of the present invention solves several key problems in the prior art in industrial applications and has achieved significant technical progress, which is specifically reflected in the following aspects:

[0093] 1. Solve the technical problems of resource allocation and interception probability optimization

[0094] When facing a complex electronic warfare environment, traditional radar systems find it difficult to simultaneously take into account the requirements of multi-target resource allocation and low interception probability. The present invention introduces a joint resource scheduling and transmission zeroing method based on reinforcement learning, and uses a deep reinforcement learning algorithm to optimize the radar transmission waveform, power allocation, and dwell time, and reduces the probability of radar signal being intercepted through transmission zeroing technology. Compared with the complexity of traditional optimization methods in dealing with multi-target and multi-constraint problems, the reinforcement learning algorithm used in the present invention can better adapt to complex environments and avoid the limitations of traditional methods when facing dynamic and multi-dimensional target optimization.

[0095] 2. Improve the detection capability of radar systems in complex electronic warfare environments

[0096] In complex electromagnetic environments, the target tracking and detection performance of existing radar systems are often significantly affected, especially when there is a large threat of electronic interference or interceptors. The present invention improves the real-time resource allocation efficiency and detection performance of radar systems in complex electronic warfare environments by adopting the proximal policy optimization (PPO) algorithm. The PPO algorithm can quickly learn and adjust during the interaction with the environment, significantly improving the stealth performance and target tracking capability of the system, ensuring that the radar system can respond quickly in complex battlefield environments with good real-time performance.

[0097] 3. Achieve long-term mission optimality and low probability of interception

[0098] Traditional radar systems often focus on short-term performance optimization during the mission, and it is difficult to ensure the optimal detection performance throughout the mission cycle. The present invention can achieve long-term optimal performance of the mission by constructing the optimization objectives of the cumulative signal-to-noise ratio and the real-time signal-to-noise ratio for the entire mission, and introducing an adaptive adjustment mechanism based on reinforcement learning. At the same time, the system effectively reduces the interception probability and improves the concealment of the radar system by optimizing power allocation and beam direction control.

[0099] 4. Real-time dynamic scheduling and transmission zeroing

[0100] In the prior art, traditional resource scheduling methods are usually based on fixed parameters or static models, which are difficult to cope with dynamic changes in complex environments. The present invention realizes real-time dynamic scheduling of resources through reinforcement learning algorithms, enabling the radar system to dynamically adjust the transmission waveform, power and dwell time according to environmental changes. At the same time, the system uses transmission zeroing technology to retain power in the target direction and achieve signal zeroing in the interceptor direction, thereby reducing the probability of being detected by enemy interceptors.

[0101] 5. Improve radar resource utilization efficiency and stealth performance

[0102] In a MIMO radar network with multiple transmitting nodes and multiple receiving nodes, how to efficiently utilize resources is always a difficult problem. The present invention can dynamically allocate radar resources and maximize their utilization through the adaptive resource allocation strategy of the reinforcement learning algorithm. At the same time, according to the actual signal-to-noise ratio and intercept probability requirements, the system improves the stealth performance of the radar system through power control and transmission zeroing technology, ensuring the concealment and security of the mission.

[0103] 6. Broad industrial application prospects

[0104] The technical solution of the present invention can be widely used in the fields of radar detection, national defense electronic warfare, drone target tracking, etc., especially in scenarios that require multi-node collaborative detection, low interception probability, and efficient resource utilization. By optimizing radar resource allocation and improving stealth performance, the present invention can be applied to the design and upgrade of modern radar systems, especially in the fields of national defense and security, with important market prospects and industrial application value.

[0105] In summary, the present invention has achieved significant technological progress while solving the problems of the prior art through reinforcement learning and transmission zeroing technology, and has provided an innovative solution for efficient covert detection and resource scheduling of radar systems in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0106] Figure 1 It is a schematic diagram of the movement of the target and the interceptor in the simulation scene of the present invention.

[0107] Figure 2 It is a graph showing changes in rewards during the training process of the present invention.

[0108] Figure 3 It is a graph showing the changes of the actions performed by the radar over time in the simulation scenario of the present invention.

[0109] Figure 4 It is a comparison diagram of the effects of the PSO algorithm and the PPO algorithm of the present invention. DETAILED DESCRIPTION

[0110] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0111] In view of the lack of dynamic adjustment capability and comprehensive consideration of radar system resources in traditional radar resource configuration methods, the present invention considers transmission zeroing and resource scheduling for joint optimization, takes the node's transmission waveform, residence time and power allocation as optimization variables, and considers the cumulative signal-to-noise ratio constraint and the low interception probability of the entire task. Finally, an optimization model for minimizing the interception probability is proposed, and then, for the above-mentioned optimization model, an algorithm based on reinforcement learning is adopted to solve it, specifically the PPO algorithm is adopted, and finally the low interception probability and resource management efficiency are improved.

[0112] The joint resource scheduling and transmission zeroing method based on reinforcement learning provided in an embodiment of the present invention comprises the following steps:

[0113] S101, establishing a system model, considering a MIMO radar network with separate transmission and reception, where multiple transmitting nodes and multiple receiving nodes cooperate to perform a target tracking task;

[0114] S102, perform problem modeling, establish a resource allocation model, define relevant variables such as transmission signal, power, signal-to-noise ratio, low interception probability, etc., and propose a target optimization function to minimize the interception probability of the entire task while satisfying the task cumulative signal-to-noise ratio and real-time signal-to-noise ratio;

[0115] S103, solves the resource allocation model and develops a deep reinforcement learning-based strategy for real-time scheduling of radar resources and transmission zeroing. This strategy optimizes radar resource usage through interactive learning with the environment, including selecting transmit waveforms, adjusting dwell time, and dynamically allocating power. At the same time, PPO is used to achieve transmission zeroing, reduce the probability of radar signals being intercepted, and improve the stealth performance of the radar system and its effective detection and tracking capabilities in complex electronic warfare environments.

[0116] In step 1 provided in an embodiment of the present invention, consider a MIMO radar network with separate transmission and reception, where there are Q transmitting nodes and Q receiving nodes, and each node is a MIMO radar that collaboratively tracks Q targets.

[0117] The second step provided in the embodiment of the present invention is to model the problem.

[0118] The matrix of the transmitted signal at time t can be expressed as:

[0119]

[0120] x t,m,q =[x t,m,q (1),x t,m,q (2),…,x t,m,q (L)] is a row vector of size L, representing the waveform transmitted by the qth antenna, with a sampling number of L. The composite signal in the θ direction can be expressed as:

[0121]

[0122] A t,m is the amplitude coefficient of the waveform matrix transmitted by radar m at time t. θ is the emission direction vector, which can be expressed as:

[0123]

[0124] Where d is the distance between the transmitting antennas and λ is the signal wavelength. The average power in the θ direction during the pulse duration can be expressed as:

[0125]

[0126] In order to achieve LPI while ensuring detection performance, the signal-to-noise ratio and intercept probability are considered. At time t, the signal-to-noise ratio of radar m to target n can be expressed as:

[0127]

[0128] Where N t,m is the cumulative number of correlations, P t,m,n G is the transmission power of radar m to target n at time t. recive is the gain of the radar receiving antenna. Noise is the noise power of the radar receiving system, which can be obtained by noise = KT r B r F r Calculate. K is the Boltzmann constant. Tr is the absolute temperature of the radar receiver. Br is the bandwidth of the radar receiver, and Fr is the noise factor of the radar receiver. β t,m,n is the attenuation, taking into account the target scattering and propagation distance, which can be expressed as:

[0129]

[0130] In the formula, σ n is the radar cross section (RCS) of the detected target n, R t,m,n is the distance between radar m and target n at time t.

[0131] For LPI performance, considering the interception probability of radar m at time t, it can be expressed as:

[0132]

[0133] Where T t,m =N t,m T pulse , residence time T t,m It can be expressed as N t,m T pulse is the pulse duration of the transmitted waveform, T j is the search time of the interceptor. The error complement function (·) is the complementary error function. p′ fa is the interceptor false alarm probability. t,m,j G is the transmission power of radar m in the direction of interceptor j. r is the gain of the interceptor receiving antenna. G P is the processing gain of the interceptor. t,m,j is the distance between radar m and interceptor j. 0 is the absolute temperature of the interception receiver. B is the bandwidth of the interception receiver, and F is the noise factor of the interception receiver. From equations (5) and (7), we can see that SNR t,m,n and p t,m N t,m ,At,m and X t,m OK, make P t,m,n Get bigger, P t,m,j In our method, the transmit waveform matrix is ​​obtained by adopting

[12] , which can be expressed as a function:

[0134] X t,m =opt(a t,m ) (8)

[0135] Where a t,m It is a parameter that controls the proportion of transmission power in each target direction. It can realize power control in different target directions and real-time transmission zero value in the interceptor direction. t,m ,A t,m and a t,m , you can use resource management to achieve better performance.

[0136] It should be noted that the above considerations are radars. For networked radars, the signal-to-noise ratio of target n at time t can be described as:

[0137]

[0138] The probability that at least one radar is intercepted at time t can be described as:

[0139]

[0140] While satisfying the mission cumulative signal-to-noise ratio and real-time signal-to-noise ratio, the interception probability of the entire mission is minimized, which can be expressed as:

[0141]

[0142] Where T is the time step of the entire task.

[0143] In step three provided in the embodiment of the present invention, the PPO algorithm is used to solve the optimization model:

[0144] In this study, we apply proximal policy optimization (PPO) to resource scheduling in a network radar system. States, observations, actions, and rewards are presented separately. The state of the mth agent is defined as:

[0145]

[0146] T is the time step. and are the two-dimensional coordinates of the target n.

[0147] is the cumulative signal-to-noise ratio of target n from time step 0 to t-1. and are the 2D coordinates of the interceptor. The state contains complete information describing the current environment, including the positions of the target and interceptor, the accumulated signal-to-noise ratio, and the time step.

[0148] The observation value of the mth agent is defined as:

[0149]

[0150] and are the angle and distance from radar m to target n at time step t, respectively. and is the angle and distance from radar m to interceptor j, t is the time step, is the cumulative signal-to-noise ratio of target n from time step 0 to t-1. The observation result is the information that the radar can obtain, and the observation result is input into the decision network to determine the action at the next moment.

[0151] The action of the mth agent is defined as:

[0152]

[0153] The action is the control of the mth radar, which describes the emission of radar m. It consists of the accumulation number N associated with the dwell time t,m , and the amplitude coefficient A associated with the nulled beam pattern t,m and the scale parameter a t,m .

[0154] The reward of the mth agent is defined as:

[0155]

[0156] Among them, r m,1 t , r m,2 t , r m,3 t are the reward for interception probability in the loss function, the constraint for cumulative signal-to-noise ratio, and the constraint for real-time signal-to-noise ratio. The min() function is used to prevent excessive rewards from affecting the behavior of the agent. Specifically, r m,1 t Designed for:

[0157] r m,1 t =-γ 1 ln(p t )-0.5 (16)

[0158] The value range is -0.5 to 0.5. 1is the interception probability reward coefficient. To implement the constraints in equation (11), the cumulative signal-to-noise ratio r is introduced m,2 t The constraint reward is designed as follows:

[0159]

[0160] It is the overall constraint for the entire task. At the end of the task, the cumulative signal-to-noise ratio is calculated. If the agent satisfies the constraint, a positive reward is given, otherwise a negative reward is given. m,3 t To achieve real-time signal-to-noise ratio constraints, the design is:

[0161]

[0162] Finally, the present invention adopts the PPO algorithm to perform resource scheduling and transmission zeroing for multiple radars and multiple targets.

[0163] The detailed execution steps of the algorithm are as follows:

[0164] 1. Initialize the policy network π and set the parameters ψ, initialize the value network V and set the parameters φ, and initialize the hyperparameters of PPO.

[0165] 2. For each training number n train From 1 to N train Perform the following steps:

[0166] 3. Initialize the playback buffer D of the mth radar m . Initialize the target and interceptor positions.

[0167] 4. For each radar m from 1 to M, perform the following steps:

[0168] 5. Get the status of the mth radar and noisy observations

[0169] 6. Calculate Actions

[0170] 7. Calculate X t,m According to equation (7), the maximum value p in the noise range is calculated t,m , calculate p according to equation (10) t According to equation (5), the minimum signal-to-noise ratio SNR in the noise range is calculated t,m,n , calculate SNR according to equation (9) t,n . Calculate the reward r according to equation (15) t .

[0171] 8. If t < T, perform the following steps:

[0172] 9. The target and interceptor fly to the next position randomly. Calculation and

[0173] 10. Otherwise:

[0174] 11. Initialize the positions of the target and interceptor.

[0175] 12. Calculate the advantage function

[0176] 13. For each radar m from 1 to M, perform the following steps:

[0177] 14. Stored in playback buffer D m middle.

[0178] 15. For each update number n update From 1 to N update Perform the following steps:

[0179] 16. For each radar m from 1 to M, perform the following steps:

[0180] 17. From D m Sample a batch of transformations in . Update the parameters of the policy network and the value network according to PPO.

[0181] 18. The algorithm ends.

[0182] The policy network π is used to control the action of the radar, and the value network V is used to estimate the cumulative benefits, which is the calculation advantage function A t,m Each radar interacts with the environment according to the policy network π, collecting data from step 2 to step 17. The policy network π and the value network V are updated from step 13 to step 18.

[0183] The radar system for joint resource scheduling and transmission zeroing based on reinforcement learning provided by an embodiment of the present invention includes:

[0184] The system model building module is used to build a networked radar system consisting of a MIMO radar with separate transmitters and receivers. Multiple transmitting nodes and multiple receiving nodes of the radar system cooperate to perform target tracking tasks.

[0185] The resource optimization model building module is used to minimize the interception probability and build an optimization model for joint resource scheduling and transmission zeroing under the conditions of satisfying the real-time signal-to-noise ratio constraints and the cumulative signal-to-noise ratio constraints, taking the radar's transmission waveform, dwell time and power allocation as optimization variables;

[0186] Resource optimization model solving module,The resource optimization model solving module adopts a reinforcement learning-based algorithm and applies the PPO algorithm to realize real-time intelligent scheduling of radar resources, and realizes transmission zeroing in a data-driven way;

[0187] The present invention applies the joint resource scheduling and transmission zeroing method based on reinforcement learning to a computer device, wherein the computer device includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the joint resource scheduling and transmission zeroing method based on reinforcement learning.

[0188] The joint resource scheduling and transmission zeroing method based on reinforcement learning provided by the application embodiment of the present invention is applied to an information data processing terminal, and the information data processing terminal is used to implement the radar system of joint resource scheduling and transmission zeroing based on reinforcement learning.

[0189] The following are two specific embodiments of the joint resource scheduling and transmission zeroing method based on reinforcement learning:

[0190] 1. Application in military radar systems

[0191] In modern electronic warfare environments, radar systems face complex electromagnetic interference and interception threats. When facing enemy electronic countermeasure systems, traditional radar systems find it difficult to reduce the risk of being intercepted while ensuring detection accuracy. The present invention optimizes radar resource allocation and signal transmission strategy through reinforcement learning and transmission zeroing technology, so that the radar can dynamically adjust the transmission waveform and power when detecting and tracking targets, and realize zeroing of the signal in the direction of the enemy interceptor, reducing the probability of the radar signal being detected.

[0192] Through reinforcement learning algorithms, the system interacts with the environment in complex electronic warfare environments and continuously optimizes the transmission waveform and power distribution.

[0193] The PPO algorithm is used to achieve stealth and low probability of interception of radar signals without sacrificing detection capability.

[0194] The radar system uses real-time transmission zeroing technology to automatically adjust radar resource allocation when facing enemy electronic interference, ensuring that the risk of being detected by interceptors is reduced while maintaining the cumulative signal-to-noise ratio.

[0195] In military exercises, the radar system trained by reinforcement learning successfully coped with complex electromagnetic interference environments, reducing the interception probability of the mission by 30% and improving the target tracking accuracy by 15%. This technology is widely used in military radar systems and can significantly improve the survivability of radar systems.

[0196] 2. Application in UAV swarm collaborative detection system

[0197] In drone group collaborative tasks, radars carried by multiple drones need to jointly perform complex target detection tasks. Traditional radar resource allocation methods are difficult to effectively cope with the dynamic changes of multiple targets and interference sources. The method of the present invention uses reinforcement learning algorithms and transmission zeroing technology to enable the radars of drones to dynamically allocate resources during collaborative detection, adjust the transmission waveform and power according to mission requirements and interference situations, and thus ensure the smooth completion of the mission.

[0198] In the UAV collaborative detection mission, the radar system uses reinforcement learning algorithms to adaptively adjust the transmission signal and power allocation to ensure resource coordination among multiple UAVs.

[0199] Through the PPO algorithm, the system can dynamically adjust the radar signal emission direction according to the current electronic environment during mission execution, and implement signal zeroing in the direction of the interceptor to improve stealth.

[0200] The system uses the cumulative signal-to-noise ratio of radar signals to evaluate the detection effect of the target, ensuring that resource scheduling can be optimized in real time during execution to maximize detection performance.

[0201] In the simulated UAV group detection mission, the radar system successfully reduced the interception probability by 40% while improving the target detection accuracy by 10% through the reinforcement learning algorithm and transmission zeroing technology of the present invention, allowing the UAV group to perform the detection mission more covertly. This technology is suitable for resource optimization scheduling in UAV cluster missions and has broad application prospects.

[0202] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. A joint resource scheduling and transmission zeroing method based on reinforcement learning, characterized in that: include: (1) Establish a system model to meet the requirements of modern electromagnetic environments for radar systems. Consider the situation in which multiple transmitting nodes and multiple receiving nodes collaborate to perform target tracking tasks in a MIMO radar network with separate transmitters and receivers. (2) Model the problem, establish a resource allocation model, define variables such as the transmission signal, power, signal-to-noise ratio, and low intercept probability, and construct a target optimization function to minimize the intercept probability of the task while satisfying the task's cumulative signal-to-noise ratio and real-time signal-to-noise ratio. This method optimizes the entire task; (3) Solve the resource allocation model and use a deep reinforcement learning algorithm to optimize radar resource usage through interactive learning with the environment, including selecting the transmit waveform, adjusting the dwell time, dynamically allocating power, achieving transmission zeroing, and reducing the probability of radar signals being intercepted; (4) The proximal policy optimization (PPO) algorithm is used to optimize the model and improve the detection and tracking capabilities of the radar system in complex electronic warfare environments.

2. The method for joint resource scheduling and transmission zeroing based on reinforcement learning according to claim 1, characterized in that: The system model described in step (1) is a MIMO radar network with separate transmission and reception, including Q transmitting nodes and Q receiving nodes, and each node is a MIMO radar.

3. The method for joint resource scheduling and transmission zeroing based on reinforcement learning according to claim 1, characterized in that: In the resource allocation model defined in step (2), the matrix of the transmitted signal can be expressed as a matrix function, which combines power control and signal-to-noise ratio constraints to achieve power control in different target directions and real-time transmission zeroing in the interceptor direction.

4. The method for joint resource scheduling and transmission zeroing based on reinforcement learning according to claim 1, characterized in that: In step (3), the PPO algorithm optimizes the radar transmission waveform and power allocation through proximal strategy optimization and training based on the interaction data between the agent and the environment. It uses the cumulative signal-to-noise ratio and interception probability as a reward mechanism to enhance the stealth performance and target tracking capability of the radar system.

5. According to the joint resource scheduling and transmission zeroing method based on reinforcement learning of claim 1, further, in step 2, the problem is modeled, and the matrix of the transmitted signal at time t can be expressed as: x t,m,q =[x t,m,q (1),x t,m,q (2),…,x t,m,q (L)] is a row vector of size L, representing the waveform transmitted by the qth antenna. The number of sampling times is L, and the synthetic signal in the θ direction can be expressed as: A t,m is the amplitude coefficient of the waveform matrix transmitted by radar m at time t, a θ is the emission direction vector, which can be expressed as: Where d is the distance between transmitting antennas, λ is the signal wavelength, and the average power in the θ direction during the pulse duration can be expressed as: In order to achieve LPI while ensuring detection performance, the signal-to-noise ratio and intercept probability are considered. At time t, the signal-to-noise ratio of radar m to target n can be expressed as: Where N t,m is the cumulative number of correlations, P t,m,n is the transmitting power of radar m to target n at time t, G recive is the gain of the radar receiving antenna, noise is the noise power of the radar receiving system, which can be obtained by noise noise = KT r B r F r Calculation, K is the Boltzmann constant, Tr is the absolute temperature of the radar receiver, Br is the bandwidth of the radar receiver, Fr is the noise coefficient of the radar receiver, β t,m,n is the attenuation, taking into account the target scattering and propagation distance, which can be expressed as: In the formula, σ n is the radar cross section (RCS) of the detected target n, R t,m,n is the distance between radar m and target n at time t; For LPI performance, considering the interception probability of radar m at time t, it can be expressed as: Where T t,m =N t,m T pulse , residence time T t,m It can be expressed as N t,m T pulse is the pulse duration of the transmitted waveform, T j is the search time of the interceptor, the error complement function (·) is the complementary error function, p′ fa is the interceptor false alarm probability, P t,m,j is the transmission power of radar m to interceptor j, G r is the gain of the interceptor receiving antenna, G P is the processing gain of the interceptor, R t,m,j is the distance between radar m and interceptor j, T0 is the absolute temperature of the intercept receiver, B is the bandwidth of the intercept receiver, and F is the noise coefficient of the intercept receiver. From equations (5) and (7), we can see that SNR t,m,n and p t,m N t,m ,A t,m and X t,m OK, make P t,m,n Get bigger, P t,m,j becomes smaller; in our method, the emission waveform matrix is ​​obtained by adopting [12], which can be expressed as a function: X t,m =opt(a t,m ) (8) Where a t,m In order to control the parameters of the transmission power ratio of each target direction, it can realize the power control of different target directions and the real-time transmission zero value of the interceptor direction. t,m ,A t,m and a t,m , resource management can be used to achieve better performance; It should be pointed out that the above considerations are radars. For networked radars, the signal-to-noise ratio of target n at time t can be described as: The probability that at least one radar is intercepted at time t can be described as: While satisfying the mission cumulative signal-to-noise ratio and real-time signal-to-noise ratio, the interception probability of the entire mission is minimized, which can be expressed as: Where T is the time step of the entire task. For the entire task, the overall performance should be considered. To this end, real-time transmission cavitation and resource scheduling based on reinforcement learning are adopted to achieve real-time low interception detection tasks through data-driven.

6. According to the joint resource scheduling and transmission zeroing method based on reinforcement learning of claim 1, further, in step 3, a method based on reinforcement learning is used to solve the optimization model: Use the PPO algorithm to solve the optimization model: In this study, we apply proximal policy optimization (PPO) to resource scheduling in a network radar system; PPO can effectively handle both discrete and continuous action spaces and is more stable to train than reinforcement or basic Actor-Critic methods; in addition, PPO solves the problem of low sample efficiency by shortening the probability ratio and reusing sampled data, which is a good choice for its sample efficiency and stability, considering that the training samples for simulating the interaction between radar and environment are time-consuming; The states, observations, actions, and rewards are presented separately, and the state of the mth agent is defined as: T is the time step, and is the two-dimensional coordinate of target n; is the cumulative signal-to-noise ratio of target n from time step 0 to t-1, and is the two-dimensional coordinate of the interceptor, and the state contains complete information describing the current environment, including the position of the target and interceptor, the cumulative signal-to-noise ratio, and the time step; The observation value of the mth agent is defined as: and are the angle and distance from radar m to target n at time step t, and is the angle and distance from radar m to interceptor j, t is the time step, The cumulative signal-to-noise ratio of target n from time step 0 to t-1; the observation result is the information that the radar can obtain, and the observation result is input into the decision network to determine the action at the next moment; For example, if the radar is close to the target, it will be chosen to transmit; if the angle between the target and the interceptor is close, the radar will limit the radiated energy towards the target to avoid leakage to the interceptor; if the cumulative signal-to-noise ratio is not enough, the radar tends to transmit more energy; The action of the mth agent is defined as: The action is the control of the mth radar, which describes the emission of radar m, and it includes the accumulation number N related to the dwell time t,m , and the amplitude coefficient A associated with the nulled beam pattern t,m and the scale parameter a t,m ; The reward of the mth agent is defined as: in, They are the reward of interception probability in the loss function, the constraint of cumulative signal-to-noise ratio, and the constraint of real-time signal-to-noise ratio. The min() function is used to prevent excessive rewards from affecting the behavior of the agent. Specifically, Designed for: The value range is -0.5 to 0.5, γ1 is the interception probability reward coefficient, and the cumulative signal-to-noise ratio is introduced to realize the constraint in formula (11). The constraint reward is designed as follows: It is the overall constraint of the entire task; if the agent satisfies the constraint, a positive reward is given, otherwise a negative reward is given; to implement the constraint in equation (11), we introduce To achieve real-time signal-to-noise ratio constraints, the design is:

7. A radar system implementing the joint resource scheduling and transmission zeroing based on reinforcement learning as described in any one of claims 1 to 6, characterized in that: The radar system with joint resource scheduling and transmission zeroing based on reinforcement learning: The system model building module is used to build a networked radar system consisting of a MIMO radar with separate transmitters and receivers. Multiple transmitting nodes and multiple receiving nodes of the radar system cooperate to perform target tracking tasks. The resource optimization model building module is used to minimize the interception probability and build an optimization model for joint resource scheduling and transmission zeroing under the conditions of satisfying the real-time signal-to-noise ratio constraints and the cumulative signal-to-noise ratio constraints, taking the radar's transmission waveform, dwell time and power allocation as optimization variables; The resource optimization model solving module,the resource optimization model solving module adopts an algorithm based on reinforcement learning,applies the PPO algorithm to realize real-time intelligent scheduling of radar resources,and realizes transmission zeroing through a data-driven approach.

8. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the joint resource scheduling and transmission zeroing method based on reinforcement learning as described in any one of claims 1-3.

9. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the joint resource scheduling and transmission zeroing method based on reinforcement learning as described in any one of claims 1 to 3.

10. An information data processing terminal, characterized in that: The information data processing terminal is used to implement the radar system with joint resource scheduling and transmission zeroing based on reinforcement learning as described in claim 7.

Images