A reinforcement learning-based method for searching the optimal communication angle of a scattering antenna

Abstract

This invention provides a reinforcement learning-based method for searching the optimal communication angle of a scattering antenna, belonging to the field of scattering communication technology. The invention models the optimal communication angle search process as a Markov random process and optimizes the search process using reinforcement learning. The steps include: periodically acquiring the communication angle and single-carrier measured signal reception strength during the optimal communication angle search process; constructing and training a reinforcement learning model; saving the training results of the reinforcement learning model; and loading the training results to search for the optimal communication angle in new scenarios. This invention can more closely approximate the optimal communication angle with lower time costs and has good adaptability.

Description

Technical Field

[0001] This invention relates to the field of scattering communication technology, and more particularly to the field of optimal communication angle alignment of scattering antennas. Background Technology

[0002] Tropospheric scattering communication systems exhibit fading characteristics in their transmission channels. Parameters such as fading rate and fading depth are related to numerous factors, including transmission distance, operating frequency, and weather conditions. During the commissioning of a tropospheric scattering communication system, to achieve optimal communication performance, it is generally necessary to perform a fine scan of the antenna's communication angle to determine the best communication angle.

[0003] During the search for the optimal communication angle, due to the fast fading characteristics of the channel, the received signal strength measured by a single carrier in a short time is randomly distributed. Therefore, the search process for the optimal communication angle is a stochastic process. To improve the accuracy of the optimal angle estimation, it is often necessary to increase the number of scans or the dwell time. By increasing the number of samples, the estimation error of the expected value of the received signal strength at different angles can be reduced, which in turn increases the time cost. Summary of the Invention

[0004] To address the trade-off between accuracy and time cost in existing technologies for finding the optimal communication angle, this invention provides a reinforcement learning-based method for searching the optimal communication angle of a scattering antenna. The method models the search process as a Markov random process and optimizes it using a reinforcement learning algorithm. By training the algorithm with historical data, the trained model can more closely approximate the optimal communication angle with lower time costs.

[0005] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows:

[0006] A reinforcement learning-based method for searching the optimal communication angle of a scattering antenna includes the following steps:

[0007] Step S1: Periodically acquire the communication angle and single-carrier measured signal received strength during the search process for the optimal communication angle of the scattering antenna;

[0008] Step S2: Construct a reinforcement learning model and train the model;

[0009] Step S3: Save the training results of the reinforcement learning model;

[0010] Step S4: Load the training results and search for the best communication angle for the new scene.

[0011] Furthermore, in step S1, during the search for the optimal communication angle of the antenna, a timer is set to acquire the real-time antenna communication angle value and the received signal strength of the single carrier measurement signal at a fixed period.

[0012] Furthermore, the specific method for training the model in step S2 is as follows:

[0013] The current state of the scattering antenna is obtained as the first state; the next action is obtained according to the action selection strategy based on the current state as input; the selected next action is executed to obtain the second state; the single-carrier measured signal received strength in the second state is used as the reward for the executed action, and the action selection strategy is updated; the above steps are repeated until the action selection strategy reaches stability.

[0014] The current state of the scattering antenna refers to the current position of the antenna and the average value of the received signal strength measured by the antenna at various angles.

[0015] The action described is to adjust the antenna communication angle, and to adjust the antenna angle in one or more adjustment steps in different directions;

[0016] The action selection strategy is trained to select different actions in different states. Its input is the current state and available actions, and its output is the expected reward obtained by selecting the action.

[0017] The second state is the state after the antenna communication angle changes after the selection action is performed, which in turn causes a change in the antenna angle and the average value of the received signal strength measured by the antenna at each angle.

[0018] The use of the single-carrier measured signal reception strength in the second state as the reward for performing the action means that after adjusting the antenna communication angle, the measured signal reception strength changes, and the short-term signal strength value after the change is used as the reward value for performing the corresponding action in the previous state. The purpose is to update the action selection strategy.

[0019] Furthermore, the specific method for training the model in step S2 is as follows:

[0020] Step 2-1: Set the minimum and maximum communication angles of the antenna, set the initial communication angle of the antenna, set the communication angle adjustment step size of the antenna, and set an array to store the average received signal strength value of the antenna at each angle.

[0021] Initialize the Q-Table, which is represented as Q(s,a), that is, the expected reward that can be obtained by performing action a in action set A under a certain state s in state set S.

[0022] Step 2-2: Select action a, which includes adjusting the angle of the antenna in one or more adjustment steps;

[0023] Initially, the Q-Table is empty. At this point, based on the current state and the elements in the action set A, an action 'a' is randomly selected. In order to utilize the learned knowledge and take the action with the largest current Q value as the basis for selecting the next action, and to avoid getting trapped in a local optimum and expand the policy search range, an ε-greedy action selection policy needs to be executed. The design principle of the ε-greedy policy is that when selecting action 'a' each time, the next action is randomly selected with probability ε, and the action with the largest current Q value is selected as the next action with probability (1-ε).

[0024] Steps 2-3: Execute action a, controlling the antenna communication angle upwards or to the next step;

[0025] Steps 2-4: Obtain the reward. Obtain the received signal strength through the device and use the received signal strength as the reward value. At this time, in order to avoid the antenna angle from exceeding the set upper and lower limits, when the current state is at the upper or lower limit, an additional penalty is added for the overstepping action.

[0026] Steps 2-5: Update the Q-Table. The function for updating the Q-value is:

[0027]

[0028] In the formula above, s represents the current state, a represents the current action, s' represents the next state, a' represents the next action, γ is the greed factor, 0 < γ < 1; Q(s,a) represents the expected maximum reward that can be obtained by taking action a in state s, R(s,a) is the immediate reward, and the reward in the next period depends on the action in the next stage.

[0029] Step 2-6: Repeat steps 2-2 to 2-5 until the Q-Table is stable.

[0030] Furthermore, in step S3, the training results are stored in the form of a binary file.

[0031] Furthermore, the specific method of step S4 is as follows:

[0032] Step 4-1: Load the trained binary file to obtain the model;

[0033] Step 4-2: Initialize the antenna communication angle;

[0034] Step 4-3: Based on the model, calculate Q(s,a) and select action a that yields the maximum expected reward in state s.

[0035] Step 4-4: Perform action a to adjust the antenna communication angle and obtain the state s after performing the action;

[0036] Steps 4-5 are repeated, including steps 4-3 and 4-4, until the preset termination condition is reached.

[0037] The beneficial effects of this invention are as follows:

[0038] 1. This invention models the search process for the optimal communication angle of the antenna as a Markov process and optimizes the search process using reinforcement learning. Simulation experiments have verified that it can approximate the optimal communication angle with fewer search steps.

[0039] 2. This invention uses a reinforcement learning algorithm, which can be trained using historical data. As data accumulates, the algorithm has self-learning capabilities, making this method more flexible and applicable to a wider range of situations. Attached Figure Description

[0040] To more clearly describe the specific implementation process and implementation examples of the technical solution of the present invention, the technical steps and implementation examples in the text are listed in the form of figures below, and the present invention will be described more clearly and in detail in conjunction with the accompanying drawings.

[0041] Figure 1 This is a flowchart of a reinforcement learning-based method for searching the optimal communication angle of a scattering antenna, as described in an embodiment of the present invention.

[0042] Figure 2 This is a comparison of simulation results between a reinforcement learning-based method for searching the optimal communication angle of a scattering antenna and a grid method in an embodiment of the present invention.

[0043] Figure 2 In the simulation, the optimal communication angle was set to 2 degrees, and the received signal strength was distributed according to Rice distribution. The reinforcement learning algorithm and the grid search method were tested using simulation data. The number of trials was 100. The horizontal axis represents the trial number, and the vertical axis represents the optimal communication angle obtained by the two algorithms in each trial. Detailed Implementation

[0044] To further illustrate the present invention, specific embodiments are described below in more detail, along with detailed operational specifications. The following examples are illustrative of the invention, and the invention is not limited to these examples.

[0045] A reinforcement learning-based method for searching the optimal communication angle of a scattering antenna is proposed. This method models the search process as a Markov stochastic process and optimizes the search process using reinforcement learning. The method includes the following steps:

[0046] Step S1: Periodically acquire the communication angle and single-carrier measured signal received strength during the search process for the optimal communication angle of the scattering antenna;

[0047] Step S2: Construct a reinforcement learning model and train the model;

[0048] Specifically, the current state of the scattering antenna is obtained as the first state; the next action is obtained according to the action selection strategy based on the current state as input; the selected next action is executed to obtain the second state; the single-carrier measured signal received strength in the second state is used as the reward for the executed action to update the action selection strategy; the above steps are repeated until the action selection strategy reaches stability.

[0049] The current state of the scattering antenna refers to the current position of the antenna and the average value of the received signal strength measured by the antenna at various angles.

[0050] The action described is to adjust the antenna communication angle, and to adjust the antenna angle in one or more adjustment steps in different directions;

[0051] The action selection strategy is trained to select different actions in different states. Its input is the current state and available actions, and its output is the expected reward obtained by selecting the action.

[0052] The second state is the state after the antenna communication angle changes after the selection action is performed, which in turn causes a change in the antenna angle and the average value of the received signal strength measured by the antenna at each angle.

[0053] The use of the single-carrier measurement signal reception strength in the second state as the reward for performing the action means that after adjusting the antenna communication angle, the measured signal reception strength changes, and the short-time signal strength value after the change is used as the reward value for performing the corresponding action in the previous state. The purpose is to update the action selection strategy.

[0054] Step S3: Save the training results of the reinforcement learning model;

[0055] Step S4: Load the training results to search for the best communication angle for the new scene.

[0056] In step S1, during the search for the optimal communication angle of the antenna, a timer is set to obtain the real-time antenna communication angle value and the received signal strength of the single carrier measurement from the relevant hardware device at fixed periodic intervals.

[0057] The specific training method for step S2 is as follows:

[0058] Step 2-1: Set the minimum and maximum communication angles of the antenna, set the initial communication angle of the antenna, set the communication angle adjustment step size of the antenna, and set an array to store the average received signal strength value of the antenna at each angle.

[0059] Initialize the Q-Table, which is represented as Q(s,a), that is, the expected reward of taking action a (a∈A) in a certain state s (s∈S);

[0060] Step 2-2, select action a (a∈A). In this invention, the action includes making one or more adjustment steps to adjust the angle of the antenna.

[0061] Initially, the Q-Table is empty. At this point, an action 'a' is randomly selected based on the current state and the elements in set A. Subsequently, as training progresses, the Q-Table is updated according to steps 2-4. To utilize learned knowledge, the action with the largest current Q-value is used as the basis for selecting the next action. However, to avoid getting trapped in local optima and to expand the policy search range, an ε-greedy action selection strategy is also required. The design principle of the ε-greedy strategy is that each time action 'a' (a∈A) is selected, the next action is randomly selected with probability ε, and the action with the largest current Q-value is selected with probability (1-ε). The value of ε can be adjusted according to the convergence speed of the Q-Table during training.

[0062] Steps 2-3: Execute action a. Depending on the selected action, control the antenna communication angle to move upwards or to the next step.

[0063] Steps 2-4: Obtain the reward by acquiring the received signal strength through the device and using it as the reward value. At this point, to prevent the antenna angle from exceeding the set upper and lower limits, an additional penalty is applied to any boundary-crossing actions when the current state is at the upper or lower limit.

[0064] Steps 2-5: Update the Q-Table. The function for updating the Q value is:

[0065]

[0066] In the formula above, s represents the current state, a represents the current action, s' represents the next state, a' represents the next action, γ is the greed factor, 0 < γ < 1. Q(s,a) represents the expected maximum reward that can be obtained by taking action a in state s, R(s,a) is the immediate reward, and the reward in the next period depends on the action in the next stage;

[0067] Steps 2-6 are repeated from steps 2-2 to 2-5 until the Q-Table stabilizes, at which point the process ends.

[0068] In step S3, the training results are stored in the form of a binary file.

[0069] The specific method for step S4 is as follows:

[0070] Step 4-1: Load the trained binary file to obtain the model;

[0071] Step 4-2: Initialize the antenna communication angle, i.e., the initial state and related parameters of the algorithm;

[0072] Step 4-3: Based on the model, calculate Q(s,a) and select action a that yields the maximum expected reward in state s.

[0073] Step 4-4: Perform action a to adjust the antenna communication angle and obtain the state s after performing the action;

[0074] Steps 4-5: Repeat steps 4-3 and 4-4 until the termination condition is met, for example, looping 100 times, or the maximum difference between the antenna angle determined by 10 consecutive actions and the average value of these 10 angles is less than 3 steps.

[0075] During the search for the optimal communication angle using a scattering antenna, the elevation and azimuth angles of the two vehicle antennas can be roughly determined based on the geographical locations of the transmitting and receiving ends and the parking angle. After determining the approximate antenna angles, fine-tuning of the antenna angles is required to find the optimal communication angle. This invention optimizes the fine-tuning process of the antenna communication angle and is applicable to both elevation and azimuth angle adjustments, using the same principle. The specific operational details of this invention will be explained below using antenna elevation angle adjustment as an example.

[0076] When using reinforcement learning algorithms to select antenna adjustment strategies, the specific process is described as follows:

[0077] Step S1: Periodically acquire communication angle and single-carrier measured signal received strength data during the search process for the optimal communication angle of the scattering antenna;

[0078] Step S2: Construct a reinforcement learning model and train the model; specifically, this includes: obtaining the current state of the scattering antenna as the first state; taking the current state as input and obtaining the next action according to the action selection policy; executing the selected next action to obtain the second state; using the single-carrier measured signal received strength in the second state as the reward for executing the action, and updating the action selection policy; repeating the above steps until the action selection policy reaches stability.

[0079] Step S3: Save the training results of the reinforcement learning model;

[0080] Step S4: Load the training results to search for the best communication angle for the new scene.

[0081] In practice, the methods for obtaining the elevation angle and single-carrier signal reception strength during the search process for the optimal communication angle of the scattering antenna are as follows:

[0082] Set a timer to obtain real-time antenna elevation angle values ​​and single-carrier measurement signal reception strength from relevant hardware devices at fixed intervals;

[0083] The minimum elevation angle of the antenna is set to angle. min =0, in degrees, the maximum elevation angle of the antenna is set to angle. max =4, unit is degrees;

[0084] Set the initial angle of the antenna elevation. current =0, in degrees; antenna elevation adjustment step = 0.1, in degrees.

[0085] Set the length to array E average Store the average received signal strength value of the antenna at each angle;

[0086] The process of searching for the optimal communication angle of the antenna involves setting the antenna's communication angle to its minimum value. min to the maximum communication angle max Adjust back and forth between them, with an adjustment step size of step;

[0087] During the process of adjusting the antenna elevation angle, the signal reception strength of the antenna at different angles is obtained, and the array E is updated. average The corresponding value in;

[0088] Due to the fading characteristics of the transmission channel in scattering communication systems, it is difficult to find the optimal communication angle during the antenna optimal communication angle search process due to the short dwell time and small number of samples. This invention specifically employs a Q-Learning reinforcement learning model to optimize the antenna optimal communication angle search process. Q-Learning includes state, action, reward, and policy, wherein:

[0089] The states include: {E l-max >E r-max Len l-max Len r-max};

[0090] Among them, E l-max For angles smaller than angle current The maximum average received signal strength across all angles;

[0091] E r-max For angles greater than angle current The maximum average received signal strength across all angles;

[0092] Len l-max For angles smaller than anglecurrent The location of the maximum average received signal strength across all angles is at a distance from angle. current The step size;

[0093] Len r-max For angles greater than angle current The location of the maximum average received signal strength across all angles is at a distance from angle. current The step size;

[0094] The actions include: [-1, 1]. The actions represent the adjustment of the antenna communication angle in the current state. The action "-1" means that the antenna angle decreases by 1 step, and the action "1" means that the antenna angle increases by 1 step.

[0095] The strategy includes: selecting different actions based on Q-Table in different states;

[0096] The rewards include: In this invention, the short-time received signal strength measured under different states is used as the reward for the system;

[0097] The algorithm training process is as follows:

[0098] Step 2-1: Initialize the Q-Table, which is represented as Q(s, a), that is, the expected reward of taking action a (a∈A) in a certain state s (s∈S).

[0099] Step 2-2, select action a (a∈A). In this invention, the action includes two actions: adjusting the pitch angle upward or adjusting the pitch angle downward.

[0100] Initially, the Q-Table is empty. At this point, an action 'a' is randomly selected based on the current state and the elements in set A. Subsequently, as training progresses, the Q-Table is updated according to steps 2-4. To utilize the learned knowledge, the action with the largest current Q-value is used as the basis for selecting the next action. However, to avoid getting trapped in local optima and to expand the policy search range, an ε-greedy action selection strategy is also required. The design principle of the ε-greedy strategy is that each time an action 'a' (a∈A) is selected, the next action is randomly selected with probability ε, and the action with the largest current Q-value is selected as the next action with probability (1-ε). The value of ε can be adjusted according to the convergence speed of the Q-Table during training.

[0101] The design principle of the ε-greedy strategy is that, each time an action a (a∈A) is selected, the next action is randomly chosen with probability ε, and the action with the largest current Q-value is selected as the next action with probability (1-ε). The value of ε can be adjusted according to the convergence speed of the Q-Table during training;

[0102] Steps 2-3: Execute action a. Depending on the selected action, control the antenna pitch angle to move upwards or to the next step.

[0103] Steps 2-4: Obtain the reward. The received signal strength is obtained through the device and used as the reward value. At this point, to prevent the antenna angle from exceeding the set upper and lower limits, an additional penalty is added for exceeding the limits when the current state is at the upper or lower limit; the reward value is set to -1000.

[0104] Steps 2-5: Update the Q-Table. The function for updating the Q value is:

[0105]

[0106] In the formula above, s represents the current state, a represents the current action, s' represents the next state, a' represents the next action, γ is the greed factor, 0 < γ < 1. Q(s,a) represents the expected maximum reward that can be obtained by taking action a in state s, R(s,a) is the immediate reward, and the reward in the next period depends on the action in the next stage;

[0107] Steps 2-6: Repeat steps 2-2 to 2-5 until the Q-Table is stable. Then, save the Q-Table to a binary file.

[0108] The optimal communication angle scanning process is as follows:

[0109] Step 4-1: Load the trained Q-Table binary file to obtain the Q-Table;

[0110] Step 4-2: Initialize the antenna elevation angle, i.e., the initial state of the algorithm and other related parameters;

[0111] Step 4-3: Calculate Q(s,a) according to the Q-Table. At this point, the ε-greedy policy is no longer executed. Instead, the action a that can obtain the maximum expected reward in state s is directly selected.

[0112] Step 4-4: Perform action a to adjust the antenna communication angle and obtain the state s after performing the action;

[0113] Steps 4-5: Repeat steps 4-3 and 4-4 until the termination condition is met, such as reaching the specified number of times (50 times).

[0114] Figure 2 The figure shows a comparison of simulation results using this method and the mesh method. As can be seen from the figure, this method can more closely approximate the optimal communication angle at a lower time cost and has better adaptability.

[0115] In summary, this invention models the search process for the optimal communication angle of an antenna as a Markov process, and models the state description, action description, and reward value description of the search process. Furthermore, it applies a reinforcement learning algorithm to optimize this process. This invention implements a reinforcement learning-based strategy for searching the optimal communication angle of a non-constant-parameter narrowband antenna, which can approximate the optimal communication angle with fewer search steps.

Claims

1. A method for searching the optimal communication angle of a scattering antenna based on reinforcement learning, characterized in that, Includes the following steps: Step S1: Periodically acquire the communication angle and single-carrier measured signal received strength during the search process for the optimal communication angle of the scattering antenna; Step S2: Construct a reinforcement learning model and train it; the specific training method is as follows: Step 2-1: Set the minimum and maximum communication angles of the antenna, set the initial communication angle of the antenna, set the communication angle adjustment step size of the antenna, and set an array to store the average received signal strength value of the antenna at each angle. Initialize the Q-Table, which is represented as: That is, the expected benefit that can be obtained by performing action a in action set A under a certain state s in state set S; Step 2-2: Select action a, which includes adjusting the angle of the antenna in one or more adjustment steps; Initially, the Q-Table is empty. At this point, based on the current state and the elements in the action set A, an action 'a' is randomly selected. In order to utilize the learned knowledge and take the action with the largest current Q value as the basis for selecting the next action, and to avoid getting trapped in a local optimum and expand the policy search range, an ε-greedy action selection policy needs to be executed. The design principle of the ε-greedy policy is that when selecting action 'a' each time, the next action is randomly selected with probability ε, and the action with the largest current Q value is selected as the next action with probability (1-ε). Steps 2-3: Perform the action Control the antenna communication angle to move upwards or to the next step; Steps 2-4: Obtain the reward. Obtain the received signal strength through the device and use the received signal strength as the reward value. At this time, in order to avoid the antenna angle from exceeding the set upper and lower limits, when the current state is at the upper or lower limit, an additional penalty is added for the overstepping action. Steps 2-5: Update the Q-Table. The function for updating the Q-value is: In the formula above, Indicates the current state. Indicates the current action. Indicates the next state. Indicates the next action. For the greed factor, ; This represents the expected maximum reward that can be obtained by taking action a in state s. The benefits are immediate, while the benefits in the next period depend on the actions taken in the next phase. Step 2-6: Repeat steps 2-2 to 2-5 until the Q-Table is stable; Step S3: Save the training results of the reinforcement learning model; Step S4: Load the training results and search for the optimal communication angle for the new scene; the specific method is as follows: Step 4-1: Load the trained binary file to obtain the model; Step 4-2: Initialize the antenna communication angle; Step 4-3: Calculate based on the model. Choose action a, which yields the maximum expected reward in state s; Step 4-4: Perform action a to adjust the antenna communication angle and obtain the state s after performing the action; Steps 4-5 are repeated, including steps 4-3 and 4-4, until the preset termination condition is reached.

2. The method for searching the optimal communication angle of a scattering antenna based on reinforcement learning according to claim 1, characterized in that, In step S1, during the search for the optimal communication angle of the antenna, a timer is set to acquire the real-time antenna communication angle value and the received signal strength of the single carrier measurement signal at a fixed period.

3. The method for searching the optimal communication angle of a scattering antenna based on reinforcement learning according to claim 1, characterized in that, In step S3, the training results are stored in the form of a binary file.

Images