A method for optimizing the position of a distributed networking radar node based on reinforcement learning

Abstract

The application discloses a kind of distributed networking radar node position optimization methods based on reinforcement learning, it is related to radar detection technical field.The application first determines the distribution range of distributed networking radar node, radar node number and system monitoring area;Then the distribution range of distributed networking radar node is encoded;Then the reinforcement learning environment of distributed networking radar node position optimization is constructed, state and action are set, distributed networking radar node position optimization reward function, optimization network structure and training parameter are designed;Optimization network is iteratively trained;Finally, the trained network is applied to radar position decision, and the optimal radar node position combination is generated.The application models the networking radar node position optimization problem from the perspective of sequence decision, utilizes the advantage of reinforcement learning in solving sequence decision problems, and can generate the optimal coverage range of radar node position combination.

Description

Technical Field

[0001] This invention relates to the field of radar detection technology, and in particular to a method for optimizing the location of distributed networked radar nodes. Background Technology

[0002] Networked radars significantly enhance detection performance, particularly increasing the detection coverage area, through their wide-area distribution and signal-level fusion processing. The location of mobile networked radar nodes affects the overall detection coverage area of ​​the system; optimizing node distribution can further enhance the system's detection capabilities.

[0003] Existing methods use traditional optimization techniques, including genetic algorithms and particle swarm optimization, to optimize the distribution of distributed network radar nodes in order to improve detection range.

[0004] In his paper "Research on Networked Radar Resource Management Algorithm," Yang Yichuan uses the particle swarm optimization algorithm to optimize node locations and improve coverage performance. Patent publication number CN110765586A presents a radar network optimization deployment method based on an improved particle swarm optimization algorithm. While such methods have low computational cost, they struggle to achieve optimal system performance. Summary of the Invention

[0005] To address the insufficient optimization capabilities mentioned above, this invention provides a distributed radar node location optimization method based on reinforcement learning. This invention models the radar node location optimization problem from a sequential decision-making perspective, leveraging the advantages of reinforcement learning in sequential decision-making problems to generate the optimal combination of radar node locations for maximum coverage.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A distributed network radar node location optimization method based on reinforcement learning includes the following steps:

[0008] Step 1: Determine the distribution range of the distributed network radar nodes, the number of radar nodes, and the system monitoring area;

[0009] Step 2: Encode the distribution range of the distributed network radar nodes;

[0010] Step 3: Construct a distributed network radar node location optimization reinforcement learning environment and set its status and actions;

[0011] Step 4: Design the reward function for optimizing the location of distributed network radar nodes, optimize the network structure and training parameters;

[0012] Step 5: Iteratively train the optimized network;

[0013] Step 6: Apply the trained optimization network to make radar position decisions and generate the optimal combination of radar node positions.

[0014] Furthermore, the specific method for step 1 is as follows:

[0015] Step 101: Considering the influence of altitude and terrain factors on radar deployment, the distribution range of distributed network radar nodes is set to an irregular shape.

[0016] Step 102: Set a rectangular system monitoring area on the right side of the distribution range; the distribution range and the monitoring area are aligned vertically, and the rightmost vertex of the distribution range intersects with the monitoring area, and only this point intersects with the monitoring area.

[0017] Step 103: Determine the number of radar nodes.

[0018] Furthermore, the specific method for step 2 is as follows:

[0019] Step 201: Divide the distribution range of the distributed network radar nodes into two parts with x = 0 as the boundary. The left side is the part where x < 0, and the right side is the part where x ≥ 0.

[0020] Step 202: Expand all positions on the left side and encode them using [-1,0); expand all positions on the right side and encode them using [0,1].

[0021] Furthermore, the specific method for step 3 is as follows:

[0022] Step 301: Select the location of the distributed networked radar as the action of reinforcement learning, with the action output being x and the action constraint being -1≤x≤1;

[0023] Step 302: The monitoring area is gridded and the signal-to-noise ratio of each grid is calculated according to the radar equation to determine the monitoring state and obtain the current system monitoring area state matrix A; the current system monitoring area state is used as the state of reinforcement learning; when no stations are deployed, the system monitoring area state is a matrix of all zeros.

[0024] Furthermore, step 4 is performed as follows:

[0025] Step 401, define the reward function;

[0026] Step 402: Design an optimized network structure that integrates a long short-term memory neural network and a proximal policy optimization algorithm. The optimized network adopts an Actor-Critic architecture, with the Actor and Critic each being an LSTM. Both networks have the same structure and contain two hidden layers.

[0027] Step 403: Design training parameters, set the initial learning rate, the maximum training length per round, and the discount factor, where the maximum training length per round is the same as the number of radar nodes.

[0028] Furthermore, the specific method for step 6 is as follows:

[0029] After training, the optimal model saved during training is used for radar deployment, and the radar deployment plan is output.

[0030] Compared with the prior art, the present invention has the following advantages:

[0031] 1. This invention models the problem of optimizing the location of networked radar nodes from the perspective of sequential decision-making. By leveraging the advantages of reinforcement learning in solving sequential decision-making problems, it completes the optimization of the location of nodes in a distributed networked radar system through steps such as environment construction and reward setting, and can generate the optimal combination of radar node locations with the best coverage.

[0032] 2. This invention provides a distributed network radar location optimization method based on reinforcement learning. In a specific embodiment, under a given scenario, the average detection power coverage of the system in 100 random distributions (before optimization) is 2457.07, and the detection power coverage area after optimization is 4663, which is an improvement of 89.78%, and the optimization effect is significant. Attached Figure Description

[0033] Figure 1 This is a flowchart of a distributed network radar node location optimization method based on reinforcement learning in an embodiment of the present invention;

[0034] Figure 2 This is a schematic diagram of the radar node distribution range in an embodiment of the present invention;

[0035] Figure 3 This is the optimized radar node distribution diagram in this embodiment of the invention;

[0036] Figure 4 This is the radar optimization reward curve in the embodiments of the present invention;

[0037] Figure 5 This refers to the coverage area of ​​the optimized radar node deployment scheme in this embodiment of the invention.

[0038] Figure 6 This is a graph showing the coverage area results of a random array experiment in an embodiment of the present invention. Detailed Implementation

[0039] The present invention will be further described below with reference to specific embodiments. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0040] A distributed network radar node location optimization method based on reinforcement learning includes the following steps:

[0041] Step 1: Determine the distribution range of the distributed network radar nodes, the number of radar nodes, and the system monitoring area;

[0042] Step 2: Encode the distribution range of the distributed network radar nodes;

[0043] Step 3: Construct a distributed network radar node location optimization reinforcement learning environment and set its status and actions;

[0044] Step 4: Design the reward function for optimizing the location of distributed network radar nodes, optimize the network structure and training parameters;

[0045] Step 5: Iteratively train the optimized network;

[0046] Step 6: Apply the trained network to make radar position decisions and generate the optimal combination of radar node positions.

[0047] Specifically, step 1 includes the following steps:

[0048] Step 101: Considering the influence of altitude and terrain factors on radar deployment, the distribution range of distributed network radar nodes is set to an irregular shape.

[0049] Step 102: Set a rectangular system monitoring area to the right of the distribution range; the distribution range and the monitoring area are aligned vertically, and the rightmost vertex of the distribution range intersects with the monitoring area, and only this point intersects with the monitoring area;

[0050] Step 103: Determine the number of radar nodes.

[0051] Step 2 includes the following steps:

[0052] Step 201: Divide the distribution range of the distributed network radar nodes into two parts with x = 0 as the boundary. The left side is the part where x < 0, and the right side is the part where x ≥ 0.

[0053] Step 202: Expand all positions on the left side and encode them using [-1,0); expand all positions on the right side and encode them using [0,1].

[0054] Step 3 includes the following steps:

[0055] Step 301: Select the location of the distributed networked radar as the action of reinforcement learning. The action output is x, and the action constraint is -1≤x≤1.

[0056] Step 302: Grid the monitoring area and calculate the signal-to-noise ratio of each grid according to the radar equations to determine the monitoring state, thus obtaining the current system monitoring area state matrix A. Use the current system monitoring area state as the state for reinforcement learning. When no stations are deployed, the monitoring area state is an all-zero matrix.

[0057] Step 4 includes the following steps:

[0058] Step 401: Define the reward function;

[0059] Step 402: Design an optimized network structure that integrates a long short-term memory neural network and a proximal policy optimization algorithm. The optimized network adopts an Actor-Critic architecture, with each Actor and Critic being an LSTM. The network structures of Actor and Critic are the same, each containing two hidden layers.

[0060] Step 403: Design training parameters, set the initial learning rate, the maximum training length per round, and the discount factor, where the maximum training length per round is the same as the number of radar nodes.

[0061] Here is a more specific example:

[0062] like Figure 1 As shown, a distributed network radar node location optimization method based on reinforcement learning specifically includes the following steps:

[0063] Step 1, determine the distribution range of the distributed network radar nodes, the number of radar nodes, and the system monitoring area, specifically including the following steps:

[0064] Step 101: Set an irregular area as the distribution range for the distributed network radar nodes, such as... Figure 2 As shown, the region is 600km long vertically and 80km wide horizontally. The upper, lower, and left boundaries are straight lines, while the right boundary is an irregular broken line.

[0065] Step 102: Set up a rectangular system monitoring area on the right side of the distribution range, with a vertical length of 600km and a horizontal width of 280km; the distribution range is aligned vertically with the system monitoring area, and the rightmost vertex of the distribution range intersects with the monitoring area, and only this point intersects with the monitoring area.

[0066] Step 103: Determine the number of radar nodes as N = 8.

[0067] Step 2 involves encoding the distribution range of the distributed network radar nodes, specifically including the following steps:

[0068] Step 201: Divide the distribution range of the distributed network radar nodes into two parts with x = 0 as the boundary. The left side is the part where x < 0, and the right side is the part where x ≥ 0.

[0069] Step 202: Expand all positions on the left side and encode them using [-1,0); expand all positions on the right side and encode them using [0,1].

[0070] Step 3: Construct a distributed network radar node location optimization reinforcement learning environment, and set the states and actions. This includes the following steps:

[0071] Step 301: Select the location of the distributed networked radar as the action of reinforcement learning. The action output is x, and the action constraint is -1≤x≤1.

[0072] Step 302: Grid the monitoring area and calculate the signal-to-noise ratio of each grid according to the radar equations to determine the monitoring state, thus obtaining the current system monitoring area state matrix A. Use the current system monitoring area state as the state for reinforcement learning. When no stations are deployed, the monitoring area state is an all-zero matrix.

[0073] Step 4 involves designing the reward function for optimizing the location of distributed network radar nodes, optimizing the network structure and training parameters, and includes the following steps:

[0074] Step 401: Define the reward function

[0075] Step 402: Design an optimized network structure that integrates a long short-term memory neural network and a proximal policy optimization algorithm. The optimized network adopts an Actor-Critic architecture, with each Actor and Critic being an LSTM. The network structures are identical, each containing two hidden layers, with 64 hidden units in each layer.

[0076] Step 403: Design training parameters, with the initial learning rate set to 1e. -3 At the 100,000th and 200,000th time steps, the time step size is reduced to 0.1 times the original size. The maximum length of each training round is N=8, the same as the number of radar nodes, and the discount factor is set to 0.99.

[0077] Step 5: Iteratively train the optimized network.

[0078] Step 6: After training, retrieve the optimal model saved during training for radar deployment, output the radar deployment plan, and plot the radar node locations as shown in the image. Figure 3 As shown.

[0079] The radar optimization reward curve during the training process is as follows: Figure 4 As shown, by saving the optimal model and deploying the radar, an optimized radar deployment scheme can be obtained, and the radar coverage area can be plotted as follows. Figure 5 As shown in the figure. Calculations show that the optimized coverage area reaches 4663. The coverage results obtained from 100 randomized station deployment tests are as follows. Figure 6As shown, the average value is 2457.07. The optimized array layout results increased the coverage area by 86.89% compared to the average of 100 random array layout results. Optimizing the location of distributed radar nodes based on reinforcement learning can significantly improve the radar coverage area.

[0080] The above description, in conjunction with specific embodiments, provides a further detailed explanation of the present invention and should not be construed as limiting the specific implementation of the invention to these descriptions. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention, without departing from the essence of the invention, and these modifications and combinations are still within the protection scope of this invention.

Claims

1. A method for optimizing the location of distributed networked radar nodes based on reinforcement learning, characterized in that, Includes the following steps: Step 1: Determine the distribution range of the distributed network radar nodes, the number of radar nodes, and the system monitoring area; Step 2: Encode the distribution range of the distributed network radar nodes; the specific method is as follows: Step 201, the distribution range of the distributed network radar nodes is... Divided into two parts by a boundary, the left side is The right side is the part. Part of; Step 202, expand all positions on the left side, using... Encode the code, expand all positions on the right side, and use... Encode; Step 3: Construct a distributed network radar node location optimization reinforcement learning environment and set its states and actions; the specific method is as follows: Step 301: The distributed network radar location selection is used as the action in reinforcement learning, and the action output is... Action constraints are ; Step 302: The monitoring area is gridded and the signal-to-noise ratio of each grid is calculated according to the radar equation to determine the monitoring state and obtain the current system monitoring area state matrix A; the current system monitoring area state is used as the state for reinforcement learning. When no stations are deployed, the system monitoring area is a matrix of all zeros. Step 4: Design the reward function for optimizing the location of distributed network radar nodes, optimize the network structure and training parameters; the specific method is as follows: Step 401, define the reward function; Step 402: Design an optimized network structure that integrates a long short-term memory neural network and a proximal policy optimization algorithm. The optimized network adopts an Actor-Critic architecture, with the Actor and Critic each being an LSTM. Both networks have the same structure and contain two hidden layers. Step 403: Design training parameters, set the initial learning rate, the maximum training length per round, and the discount factor, where the maximum training length per round is the same as the number of radar nodes; Step 5: Iteratively train the optimized network; Step 6: Apply the trained optimization network to make radar position decisions and generate the optimal combination of radar node positions.

2. The method for optimizing the location of distributed networked radar nodes based on reinforcement learning according to claim 1, characterized in that, The specific steps for step 1 are as follows: Step 101: Considering the influence of altitude and terrain factors on radar deployment, the distribution range of distributed network radar nodes is set to an irregular shape. Step 102: Set a rectangular system monitoring area on the right side of the distribution range; the distribution range and the monitoring area are aligned vertically, and the rightmost vertex of the distribution range intersects with the monitoring area, and only this point intersects with the monitoring area. Step 103: Determine the number of radar nodes.

3. The method for optimizing the location of distributed networked radar nodes based on reinforcement learning according to claim 2, characterized in that, The specific steps for step 6 are as follows: After training, the optimal model saved during training is used for radar deployment, and the radar deployment plan is output.

Images