High-dynamic scene beam tracking method and system based on deep reinforcement learning

Abstract

The invention discloses a high-dynamic scene beam tracking method and system based on deep reinforcement learning, and belongs to the technical field of wireless communication. The method comprises the following steps: constructing a Markov decision process model, and defining a state space, an action space and a reward function based on pilot signal strength; establishing a deep Q network optimization mechanism, adopting a neural network to approach a Q value function, introducing a target network and random experience playback stable training, and balancing exploration and utilization through a greedy strategy; a priority experience playback mechanism is introduced, sample importance is quantified based on a time sequence difference error, and high-value experience is sampled preferentially; and establishing a beam misalignment detection and adjustment mechanism, setting a misalignment criterion by taking a half-power beam width as a reference, triggering a local complementary measurement process when a tracking error exceeds a threshold value, and repositioning an optimal direction. The system comprises corresponding modules. The method can effectively cope with the beam tracking challenge in a high dynamic scene, improves the tracking precision, real-time performance and robustness, and is suitable for high-speed mobile communication environments such as an unmanned aerial vehicle group.

Description

Technical Field

[0001] This invention belongs to the field of wireless communication technology, specifically relating to a high dynamic scene beam tracking method and system based on deep reinforcement learning. Background Technology

[0002] With the rapid development of UAV technology, high-speed UAV swarms are increasingly widely used in various fields. These scenarios place stringent demands on the high speed, low latency, and high reliability of communication systems. Millimeter-wave massive MIMO technology, with its ultra-large bandwidth and high spectral efficiency, has become the core solution supporting high-speed UAV swarm communication. Beam tracking, as a key technology for maintaining link stability, needs to adapt in real time to the drastic angle changes caused by the high-speed maneuvers of the UAV swarm. However, in highly dynamic UAV swarm scenarios, beam tracking faces many practical challenges. The high-speed movement of UAVs leads to the beam pointing needing to respond quickly to drastic angle changes. Complex interactions between nodes within the swarm and external environmental interference further exacerbate channel instability. At the same time, the collaborative communication of a large number of nodes places even more stringent demands on the real-time performance and robustness of beam tracking. These factors together make it difficult for existing communication systems to meet the practical application requirements in terms of ensuring continuous link stability and coping with dynamic topology changes, becoming a major bottleneck restricting the improvement of communication performance in highly dynamic UAV swarms. Therefore, it is urgent to design an efficient and reliable beam tracking method for these complex scenarios.

[0003] In the field of high-dynamic crowd beam tracking, existing solutions can be broadly classified into two categories, each with its own technical approach and applicable scenarios, but both suffer from unavoidable limitations. One category is traditional model-driven filtering algorithms, with the Extended Kalman Filter (EKF) as a typical example, and also including derivative methods such as the Unscented Kalman Filter (UKF) and Particle Filter (PF). The core logic of these algorithms is to construct accurate state transition models and observation models, and then recursively optimize them using historical state estimation results and real-time observation data to achieve continuous tracking of dynamic changes in the beam angle. Their advantage lies in relying on rigorous mathematical modeling, which allows for high tracking accuracy with relatively low computational complexity when matching the state model with the actual scene. Therefore, they were widely used in early static or low-dynamic scenarios. However, the performance of these methods is highly dependent on the accuracy of the model. In high-dynamic crowd scenarios, the terminal motion state often exhibits strong nonlinear characteristics, and the pre-set state transition model is prone to mismatch.

[0004] Another category is data-driven reinforcement learning methods, encompassing traditional Q-learning and deep reinforcement learning. These methods do not rely on precise mathematical models but instead construct Markov decision processes to transform the beam tracking problem into a dynamic interaction of state, action, and reward. The state typically includes information such as the current beam direction and received signal strength; the action is defined as the step size and direction of beam adjustment; and the reward is designed based on the improvement in signal quality. The agent accumulates experience through continuous interaction with the environment, gradually learning the optimal beam adjustment strategy. Traditional Q-learning uses a table to store the value (Q-value) of state-action pairs, achieving stable convergence in small-scale, low-dimensional scenarios. However, in highly dynamic group scenarios, as the group size increases, the state space grows exponentially, increasing the dimensionality of the table storage and causing a sharp decline in Q-value update efficiency, making it difficult to meet real-time tracking requirements. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a beam tracking method and system for high dynamic scenes based on deep reinforcement learning, which addresses the shortcomings of the prior art. This method solves the technical problems of low tracking accuracy, poor real-time performance and insufficient robustness caused by model mismatch and state space explosion in traditional beam tracking methods in high dynamic scenes.

[0006] The present invention adopts the following technical solution: A beam tracking method for high dynamic scenes based on deep reinforcement learning includes the following steps: S1. Process the pilot signal strength received from the communication system, construct a Markov decision process model, construct a state space based on the pilot signal strength, design a discrete action function including beam angle adjustment step size, and use the ratio of received signal strength as the reward function to obtain a dynamic decision process model. S2. Construct a deep Q-network optimization mechanism to optimize the dynamic decision-making process model using a deep Q-network. This involves using a two-layer fully connected neural network to approximate the Q-value function, introducing a target network and a random experience replay mechanism to stabilize the training process, balancing exploration and utilization through a greedy strategy, and dynamically adjusting the learning rate to accelerate convergence. The training objective is to minimize the Bellman residual loss function, resulting in the trained deep Q-network model. S3. Optimize the experience replay mechanism in the trained deep Q network model by introducing a priority experience replay mechanism, quantifying the importance of experience samples based on temporal difference error, prioritizing the sampling of high-value samples and introducing importance weight correction, and initializing new samples with the maximum temporal difference error to obtain the optimized training mechanism. S4. Perform beam misalignment detection and adjustment processing on the output of the trained deep Q network model, and establish a beam misalignment detection and adjustment mechanism. The misalignment criterion is defined with half-power beamwidth as the reference. When the tracking error exceeds the threshold, the local beam supplementation process is triggered. The optimal direction is repositioned by combining the surrounding beam signal strength to obtain the adjusted beam direction, which is used to maintain link stability.

[0007] Preferably, in step S1, the state space includes the received signals from the current time slot and the previous time slot, and the state space represents all states accessible to the agent during the interaction; the action space is designed to be uniformly distributed around the maximum angle change range, and the action space A is represented as:

[0008] in, For the maximum angle change, For the step length of the movement, For the number of actions, For action, For action indexing; The reward function is constructed based on the changing trend of the received signal strength. Represented as:

[0009] in, and These are the lower and upper thresholds, respectively. The number of pilot signals before the action in each time slot. The number of pilot symbols in each time slot, In the first The first time slot The received signal of one pilot symbol, The upper limit threshold, This is the lower threshold.

[0010] Preferably, the maximum angle change Calculated based on the mobile terminal's maximum speed and minimum distance, and expressed as:

[0011] in, For the maximum speed of the mobile terminal, For the duration of the time slot, Minimum distance; the action step size Represented as:

[0012] in, This is the output layer.

[0013] Preferably, in step S2, the deep Q-network includes an input layer, a hidden layer, and an output layer, wherein the input layer includes the layer before the current time slot. Each pilot symbol signal and the previous time slot There are 1 signal, and the hidden layer includes a first hidden layer and a second hidden layer. The network size of the first hidden layer is 1. The size of the second hidden layer network is The output layer outputs the Q-values ​​of all possible actions; the target network has the same structure as the main network, and the weights of the main network are copied to the target network every C training iterations.

[0014] Preferably, the dynamic adjustment of the greedy strategy is based on the current round number and the total number of rounds, and the exploration rate. Represented as:

[0015] in, This is the current round number. This represents the total number of rounds.

[0016] Preferably, in step S3, the sampling probability of the experience sample in the priority experience replay mechanism is controlled by the priority and hyperparameters determined by its temporal difference error, as expressed as:

[0017] in, For empirical samples In hyperparameters Priority under control For the sample In hyperparameters Priority under control; The priority Based on time difference error The calculation is expressed as:

[0018] in, It is a constant and greater than 0.

[0019] Preferably, the timing difference error for:

[0020] in, As a reward, As a discount factor, Current state - action value The next state-action value output by the target network.

[0021] Preferably, in step S4, the misalignment criterion is based on the root mean square error of beam tracking, defined as half the half-power beamwidth, and expressed as:

[0022] in, The half-power beamwidth is approximately: ; This refers to the number of receiving antennas; when the tracking error exceeds... At that time, the beam reinitialization process is triggered.

[0023] Preferably, the beam reinitialization process uses the compressed sensing spectrum estimation method to reposition the direction of arrival and reset the experience pool and network weights of the deep Q network to start a new round of tracking iteration.

[0024] Secondly, embodiments of the present invention provide a high-dynamic scene beam tracking system based on deep reinforcement learning, comprising: The state module processes the pilot signal strength received from the communication system, constructs a Markov decision process model, builds a state space based on the pilot signal strength, designs a discrete action function including beam angle adjustment step size, and uses the ratio of received signal strength as the reward function to obtain a dynamic decision process model. The optimization module establishes a deep Q-network optimization mechanism to perform deep Q-network optimization on the dynamic decision-making process model. It uses a two-layer fully connected neural network to approximate the Q-value function, introduces a target network and a random experience replay mechanism to stabilize the training process, balances exploration and utilization through a greedy strategy, and dynamically adjusts the learning rate to accelerate convergence. The training objective is to minimize the Bellman residual loss function to obtain the trained deep Q-network model. The experience module optimizes the experience replay mechanism in the trained deep Q-network model by introducing a priority experience replay mechanism, quantifying the importance of experience samples based on temporal difference error, prioritizing the sampling of high-value samples and introducing importance weight correction, and initializing new samples with the maximum temporal difference error to obtain an optimized training mechanism. The detection module performs beam misalignment detection and adjustment processing on the output of the trained deep Q network model, and establishes a beam misalignment detection and adjustment mechanism. The misalignment criterion is defined based on the half-power beamwidth. When the tracking error exceeds the threshold, a local beam re-measurement process is triggered. The optimal direction is repositioned by combining the surrounding beam signal strength to obtain the adjusted beam direction, which is used to maintain link stability.

[0025] Thirdly, a computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the above-described high dynamic scene beam tracking method based on deep reinforcement learning.

[0026] Fourthly, embodiments of the present invention provide a computer-readable storage medium including a computer program, which, when executed by a processor, implements the steps of the above-described high dynamic scene beam tracking method based on deep reinforcement learning.

[0027] Fifthly, a chip includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the above-described high dynamic scene beam tracking method based on deep reinforcement learning.

[0028] In a sixth aspect, embodiments of the present invention provide an electronic device, including a computer program, which, when executed by the electronic device, implements the steps of the above-described high dynamic scene beam tracking method based on deep reinforcement learning.

[0029] Compared with the prior art, the present invention has at least the following beneficial effects: A high-dynamic scene beam tracking method based on deep reinforcement learning is proposed. Step S1 constructs a Markov decision process model, transforming the beam tracking problem into a reinforcement learning task. Pilot signal strength is used to define the state space, action space, and reward function, enabling the system to adapt to dynamic environmental changes and avoiding the dependence on precise mathematical models in traditional model-driven methods. Step S2 employs a Deep Q-Network (DQN) optimization mechanism, approximating the Q-value function through a neural network, introducing a target network and random experience replay to stabilize the training process, using a greedy strategy to balance exploration and utilization, dynamically adjusting the learning rate, accelerating convergence, and improving learning efficiency and stability, especially performing well in high-dimensional state spaces. Step S3 introduces a Priority Experience Replay (PER) mechanism, quantifying sample importance based on temporal difference error (TD error), prioritizing the sampling of high-value experiences, and combining importance weight correction to reduce training bias and improve sample utilization efficiency and model generalization ability. Step S4 establishes a beam misalignment detection and adjustment mechanism, setting a misalignment criterion based on half-power beamwidth. When the tracking error exceeds a threshold, a local retest process is triggered to reposition the optimal direction, ensuring the continuity and reliability of the link. Overall, the method of this invention solves the real-time performance, accuracy, and robustness issues of beam tracking in highly dynamic scenarios through a data-driven approach. It is suitable for high-speed mobile environments such as UAV swarms and has stronger adaptability and anti-interference capabilities compared to traditional methods.

[0030] Furthermore, the state space includes the received signals from the current time slot and the previous time slot, effectively capturing the temporal correlation of terminal movement and enabling the agent to understand dynamic environmental changes. The action space is designed as a uniform distribution around the maximum angle change range. By parameterizing the action step size and number, the comprehensiveness and adjustability of actions are ensured, covering the possible angle change range and avoiding tracking blind spots. The reward function is constructed based on the ratio of received signal strength and sets upper and lower thresholds to mitigate measurement fluctuations caused by noise and interference, guiding the agent to optimize signal quality and improving the stability and convergence speed of the learning strategy. These features collectively enhance the accuracy and efficiency of beam tracking, enabling the system to quickly respond to angle changes in high-speed moving scenarios, reducing tracking errors and improving communication link quality.

[0031] Furthermore, the maximum angle change is derived based on the mobile terminal's maximum speed and minimum distance, enabling the motion space to adapt to the motion characteristics of dynamic scenes and ensuring that the beam adjustment range covers the terminal's maximum possible displacement. The motion step size is calculated using a uniform distribution formula, achieving a reasonable division of the motion space and allowing for fine-tuning of the beam direction, avoiding over- or under-adjustment issues. This parametric design improves the method's flexibility and practicality, maintaining optimal performance in scenarios with different speeds and altitudes, and enhancing the system's adaptability and reliability in high-speed maneuvering environments.

[0032] Furthermore, the input layer integrates pilot signals from the current and previous time slots to capture the temporal evolution of the channel state, providing rich features for the network. The hidden layer employs a two-layer fully connected structure, using the ReLU activation function to enhance nonlinear modeling capabilities, enabling the network to learn complex environmental mappings. The output layer outputs the Q-values ​​of all possible actions, supporting the agent in selecting the optimal beam adjustment strategy. The introduction of the target network stabilizes the training process, reducing Q-value fluctuations through lag updates and improving convergence and performance. This network design optimizes learning efficiency, achieving fast and accurate beam tracking in high-speed dynamic scenarios.

[0033] Furthermore, the dynamic adjustment rules for the exploration rate of the greedy strategy were clarified, effectively balancing the model's exploration and utilization capabilities. A high exploration rate is set in the early stages of training, allowing the agent to fully explore the action space and uncover potential optimal strategies. In the mid-stage, the exploration rate decreases linearly, gradually transitioning to primarily utilizing learned strategies. In the later stages, a fixed low exploration rate ensures the stability of the strategy. This dynamic adjustment method avoids local optima caused by insufficient exploration in the early stages and prevents excessive exploration in the later stages from affecting tracking accuracy, significantly improving the model's convergence speed and final performance, enabling the technical solution to quickly adapt to the complex changes in highly dynamic scenarios.

[0034] Furthermore, a priority experience replay mechanism is introduced. Based on the temporal difference error (TD error), the importance of experience samples is quantified, and samples with high TD errors are sampled first, focusing on key experiences and accelerating the learning speed. By controlling the sampling probability through hyperparameters and combining it with importance weight correction, sampling bias is reduced, and sample diversity is maintained. This mechanism improves sample utilization efficiency, enabling the network to converge to the optimal strategy faster and improving the accuracy and real-time performance of beam tracking.

[0035] Furthermore, the calculation method of temporal difference error was clarified, providing a scientific quantitative basis for the priority experience replay mechanism. The temporal difference error, calculated by combining immediate reward, discount factor, and Q-values ​​of the current and next states, accurately reflects the value of experience samples, providing an objective standard for sample priority ranking. This calculation logic aligns with the value assessment principle of reinforcement learning, ensuring the accurate identification of high-value samples, enabling the priority experience replay mechanism to function effectively, thereby improving the efficiency and accuracy of model training and guaranteeing the tracking performance of the technical solution in highly dynamic scenarios.

[0036] Furthermore, a beam misalignment criterion is defined, setting a root mean square error threshold based on the half-power beamwidth to scientifically and rationally determine tracking failure. When the error exceeds the threshold, an adjustment mechanism is triggered to ensure the beam remains aligned with the target, avoiding link interruption. This criterion improves the system's reliability and continuity, making it suitable for highly dynamic scenarios.

[0037] Furthermore, the specific implementation method of beam reinitialization was clarified to ensure rapid recovery after beam misalignment. A compressed sensing spectrum estimation method was adopted to reposition the incoming beam direction, achieving high positioning accuracy and fast response speed. Resetting the experience pool and network weights avoided interference from invalid historical experience, enabling rapid convergence of the new tracking cycle. This design solves the problem of difficult rapid recovery after beam misalignment, significantly improving the fault tolerance and robustness of the technical solution, ensuring that even in highly dynamic scenarios, stable links can be quickly re-established despite beam misalignment.

[0038] It is understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0039] In summary, the method of this invention combines Markov decision processes and deep Q-network (DQN) frameworks. By constructing a dynamic state space, optimizing action strategies and experience playback mechanisms, it achieves accurate tracking of the angle of high-speed mobile terminals and improves anti-interference capability and real-time performance in complex environments.

[0040] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0041] Figure 1Here is a scene diagram showing the application of the method of the present invention. Figure 2 Here is a flowchart of the DQN algorithm; Figure 3 This is a schematic diagram of the DQN beam tracking process; Figure 4 Network structure diagram; Figure 5 Random experience replay graph; Figure 6 The real-time tracking results of DQN are shown in the figure under the conditions of N=64, SNR=20dB, and T=200. Figure 7 The graph shows the changes in RMSE of DQN and EKF under different SNR conditions when the number of antennas is fixed at N=64 and T=200. Figure 8 The following is an analysis chart of the tracking experiment results with initial angle estimation errors set to 0~2° respectively; Figure 9 A schematic diagram of a computer device provided in an embodiment of the present invention; Figure 10 This is a block diagram of a chip provided according to an embodiment of the present invention.

[0042] Among them, 60. Computer equipment; 61. Processor; 62. Memory; 63. Computer program; 600. Electronic device; 610. Processing unit; 620. Storage unit; 6201. Random access memory unit; 6202. Cache memory unit; 6203. Read-only memory unit; 6204. Program / utility; 6205. Program module; 630. Bus; 640. Display unit; 650. Input / output interface; 660. Network adapter; 700. External device. Detailed Implementation

[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] In the description of this invention, it should be understood that the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0045] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0046] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Additionally, the character " / " in this invention generally indicates that the preceding and following objects have an "or" relationship.

[0047] It should be understood that although terms such as first, second, third, etc., may be used in the embodiments of the present invention to describe the preset range, these preset ranges should not be limited to these terms. These terms are only used to distinguish the preset ranges from one another. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0048] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0049] The accompanying drawings illustrate various structural schematic diagrams according to embodiments disclosed in this invention. These drawings are not to scale, and some details have been enlarged for clarity, and some details may have been omitted. The shapes of the various regions and layers shown in the drawings, as well as their relative sizes and positional relationships, are merely exemplary and may deviate from reality due to manufacturing tolerances or technical limitations. Furthermore, those skilled in the art can design regions / layers with different shapes, sizes, and relative positions as needed.

[0050] This invention provides a beam tracking method for high-dynamic scenarios based on deep reinforcement learning. Combining the environmental characteristics of high-dynamic movement of unmanned swarms, complex electromagnetic interference, and real-time communication requirements, it focuses on the design and implementation of beam tracking technology based on data-driven reinforcement learning methods to meet the requirements of accurate beam tracking, anti-interference and stable communication, and low-latency link maintenance in high-speed maneuvering scenarios, thereby improving the overall stability, resilience, and scalability of the network.

[0051] Please see Figure 1 This invention provides a beam tracking technology using a millimeter-wave massive MIMO system for high-speed mobile terminal scenarios such as drone swarms, and for BS deployment. One antenna, deployed at the MS end Both the BS and MS use ULA arrays for their antenna arrays, with the distance between adjacent elements being... satisfy , This represents half the wavelength; one radio frequency chain is dedicated to beam tracking to avoid interference with the data stream transmission. Due to the sparsity of millimeter-wave channels, for no loss of generality, the BS and MS channels are assumed to be Loss-of-Sight (LoS) single-path channels.

[0052] This invention combines the DQN decision-making module with the PER mechanism and introduces dual-threshold misalignment detection to ensure link continuity. After the algorithm starts, agent initialization is performed first, using random beam scanning to obtain initial signal strength, interference characteristics, and reward values, which serve as the initial state input for the deep Q-network. During the iterative process, the system performs DQN updates based on the observed signal in the current beam direction, i.e., combining the predicted Q-value with the actual observed reward to dynamically learn the optimal beam adjustment strategy.

[0053] Specifically, the decision-making module generates beam angle adjustment actions based on a preset greedy strategy. After the angle deflection is executed by the RF front-end, new received signal strength, channel matrix, and interference parameters are collected. Combined with the PER mechanism, empirical samples are prioritized for sampling and network parameter updates according to the timing differential error, thereby achieving adaptive tracking optimization of the target angle. Subsequently, the beam misalignment detection module is entered. The system monitors the received signal-to-noise ratio (SNR) and angle estimation error of the current beam in real time. If the SNR is lower than the link maintenance threshold or the angle error exceeds 1 / 2 of the half-power beamwidth, it is considered that the current beam has left the effective communication main lobe region, triggering the beam re-initialization process. The direction of arrival is repositioned using the compressed sensing spectrum estimation method, and the DQN experience pool and network weights are reset, thereby starting a new round of tracking iteration.

[0054] Please see Figure 2 This invention discloses a high-dynamic scene beam tracking method based on deep reinforcement learning, comprising the following steps: S1. Construct a Markov decision process model, build a state space based on pilot signal strength, design a discrete action function that includes beam angle adjustment step size, and use the ratio of received signal strength as the reward function to characterize the dynamic decision process of beam tracking. The beam tracking problem is transformed into a problem that reinforcement learning can understand using Markov decision modules. Pilot and tracking cycle design is explained. The AoA expression for the Loss of Sight (LoS) path is defined as follows: (1) According to formula 1 AoA in each time slot Simplify the channel vector to ;in, Indicates the complex channel gain. It is path loss. This indicates small-scale fading.

[0055] Path loss It is given by the following formula: (2) in, Reference distance Path loss at the location, and It is the first One time slot MS coordinate, It is the path loss factor; Following the first-order autoregressive model ,in , It is the AR coefficient. It is process noise, reflecting model uncertainty.

[0056] MS transmits in each time slot pilot symbols ,satisfy The BS end performs beamforming, and the received signal is: (3) The receive beam vector is set as follows: (4) in, It is the array response vector.

[0057] The first step will be before One symbol is used to perceive the environment, the second step is in the... After each symbol, a reward is calculated based on the received signal, and an action is taken, specifically as follows: Figure 3 As shown.

[0058] Therefore, in the first Each time slot, action In the symbol What was done afterward, therefore, the former Each symbol uses the estimated angle from the previous time slot: (5) Follow-up The estimated angle is updated after each symbol uses an action: (6) The state space is all the states that the agent can access during the interaction process, and the action space is designed to be uniformly distributed around the maximum range of angular change.

[0059] The action space encompasses all policy options that an agent can implement during interaction. In beam tracking tasks, the agent completes angle tracking by switching beam indices to adapt to changes in the target angle.

[0060] Action space Designed for a uniform distribution around the range of maximum angular variation: (9) Among them, the step length of the movement Defined as: (10) Maximum angle change Calculated based on movement speed and minimum distance: (11) Introducing parameters into motion space design To cover the maximum possible angular changes of the mobile terminal, a step size is determined. The settings can be flexibly adjusted according to specific application requirements.

[0061] The reward function in the environment is defined as the immediate feedback obtained by the agent after taking a certain action, used to guide the iterative update of the state-action values ​​in the Q-table. To guide the agent in learning a strategy to maximize the received signal power, a reward function is constructed based on the trend of changes in the received signal strength. Its expression is as follows: (12) in, and The lower and upper thresholds are used to mitigate measurement fluctuations caused by noise and interference. The numerator represents the total energy of the received signal after the action is performed, while the denominator represents the signal strength before the action. The ratio of the two is used to characterize the impact of the action on signal quality.

[0062] S2. A deep Q-network optimization mechanism is built, which uses a two-layer fully connected neural network to approximate the Q-value function. A target network and a random experience replay mechanism are introduced to stabilize the training process. A greedy strategy is used to balance exploration and utilization, and the learning rate is dynamically adjusted to accelerate convergence. The training objective is to minimize the Bellman residual loss function. The network input is a state vector. It consists of the received signals from the current time slot and the previous time slot, to capture the AoA changes caused by terminal movement. To facilitate neural network processing, this complex signal is separated into real and imaginary parts to form a real-valued vector input layer. The network structure is as follows: Figure 4 As shown in Table 1, the network parameters are as follows.

[0063] Table 1 Network structure parameters

[0064] A network consists of the following parts: Input layer Including the time slot before the current time slot Each pilot symbol signal and the previous time slot The signals are weighted to obtain the state representation.

[0065] Hidden layer: Two fully connected layers, the first layer has a size of The second layer is Each layer employs the ReLU activation function to enhance nonlinear modeling capabilities.

[0066] Output layer Outputs the Q value of all possible actions, i.e., the beam angle adjustment magnitude, for subsequent action selection.

[0067] To improve training stability and sample utilization efficiency, a random experience replay mechanism is introduced into the DQN algorithm. The agent records the state transition quadruples in each time slot. Store to playback buffer During training, mini-batches of samples are uniformly sampled from the replay buffer to update network parameters. Additionally, a target network is used to mitigate changes in the target value: every [percentage missing]. In each training iteration, the weights of the main network are copied to the target network. The training objective is to minimize the Bellman residual loss function. (13) in, Indicates the target network. This is the discount factor.

[0068] Traditional Q-learning algorithms have an action value function Overestimation can negatively impact policy convergence. To mitigate this issue, we employ a method of delayed target network updates and... A combination of greedy strategies: setting a high exploration rate in the early training phase. Then it gradually decreased to This strategy allows the agent to fully explore the benefits of different actions and stably converge to a better strategy in the later stages. Furthermore, by combining experience replay with a fixed target network, the Q-value avoids drastic fluctuations during training, thus effectively reducing overestimation bias.

[0069] (14) in, This is the current round number. This represents the total number of rounds.

[0070] S3. Introduce a priority experience replay mechanism, quantify the importance of experience samples based on time-series difference error (TD), prioritize the sampling of high-value samples and introduce importance weight correction, and use the maximum TD error to initialize new samples; The sampling probability of empirical samples is controlled by the priority and hyperparameters determined by their TD error.

[0071] To improve the efficiency of utilizing experience samples and reduce the temporal correlation between samples, a random experience replay mechanism is introduced during training. Specifically, each round of agent-environment interaction generates state transition samples, which are stored in an experience replay buffer. During the neural network training phase, the system randomly samples a small batch of samples from this buffer with uniform probability to construct the training batch. Subsequently, the network calculates the corresponding gradient information through multiple rounds of forward and backward propagation, and averages the gradients of multiple samples to complete the parameter update. This mechanism effectively breaks the temporal dependence of samples, improving the stability and generalization ability of training. The overall workflow of this process is as follows: Figure 5 As shown.

[0072] Conventional uniform random sampling strategies often struggle to efficiently select the most valuable experience samples for training in reward-sparse environments, leading to key experiences remaining unused for extended periods or even being overwritten and discarded without being sampled. To improve sampling efficiency and learning performance, priority experience replay has been proposed as an enhancement mechanism. The core idea of ​​PER (Persistent Experience Replay) is that the agent should prioritize key experiences that significantly impact the learning process. To this end, the algorithm introduces temporal difference error (TD) as a measure of sample importance, giving samples with larger TD errors a higher probability of being sampled, thereby accelerating the network's learning of key experiences. However, relying solely on TD error for sampling may lead to decreased sample diversity and reduced model generalization ability. To mitigate this problem, PER typically combines random sampling mechanisms with importance sampling correction strategies, improving learning efficiency while preserving as much experience distribution diversity as possible to reduce training bias and improve convergence performance. The experience samples in the experience pool are (…). The calculation of TD error can be expressed as: (14) Sampling TD errors solely based on a greedy strategy can lead to a series of problems. In practice, a hybrid strategy combining greedy and random sampling is often used. This approach preserves the TD error's guiding nature while introducing a degree of uniformity, ensuring that all empirical samples have a non-zero sampling probability, thereby improving training stability and generalization ability. Specifically, empirical samples... The sampling probability is determined by the priority of its TD error. and hyperparameters Control, as shown in formula (15): (15) in, Indicates priority, when When the sampling time is equal to the time, it is considered uniform sampling.

[0073] Priority as follows: (16) in, It is a constant and greater than 0, used to ensure that empirical samples have the possibility of being sampled during the sampling process.

[0074] Furthermore, to prevent newly added experiences from remaining in a state of low sampling probability for an extended period due to unknown initial TD errors, a maximum TD error initialization strategy is introduced into the priority experience replay mechanism. Compared to setting the initial priority of new samples to a constant or zero, this method is more conducive to ensuring that potentially high-value experiences can be learned in a timely manner, and is particularly suitable for scenarios where new information is continuously introduced in dynamic environments.

[0075] S4. Establish a beam misalignment detection and adjustment mechanism, define a misalignment criterion based on half-power beamwidth, and trigger a local beam remeasurement process when the tracking error exceeds the threshold, and reposition the optimal direction by combining the surrounding beam signal strength.

[0076] Define the root mean square error criterion for beam failure, i.e., RMSEBF is half the half-power beamwidth: (17) in, The half-power beamwidth is approximately expressed as: (18) When tracking is effective, the RMSE of AoA tracking should be less than the RMSEBF. Because the DQN algorithm uses discrete actions, its AoA tracking RMSE has a theoretical lower bound.

[0077] In the optimal case, given the actual AoA and discrete action space, the lower bound of RMSE can be considered the theoretical limit of maximum performance. Since there is no additional information about the mobile terminal's location, assume the optimal tracking error is within the interval... Uniformly distributed within, among which The fixed step size.

[0078] Therefore, the lower bound of RMSE for AoA tracking is calculated as follows: (19) In another embodiment of the present invention, a high dynamic scene beam tracking system based on deep reinforcement learning is provided. This system can be used to implement the above-mentioned high dynamic scene beam tracking method based on deep reinforcement learning. Specifically, the high dynamic scene beam tracking system based on deep reinforcement learning includes a state module, an optimization module, an experience module, and a detection module.

[0079] The state module processes the pilot signal strength received from the communication system, constructs a Markov decision process model, builds a state space based on the pilot signal strength, designs a discrete action function including beam angle adjustment step size, and uses the ratio of received signal strength as the reward function to obtain a dynamic decision process model. The optimization module establishes a deep Q-network optimization mechanism to perform deep Q-network optimization on the dynamic decision-making process model. It uses a two-layer fully connected neural network to approximate the Q-value function, introduces a target network and a random experience replay mechanism to stabilize the training process, balances exploration and utilization through a greedy strategy, and dynamically adjusts the learning rate to accelerate convergence. The training objective is to minimize the Bellman residual loss function to obtain the trained deep Q-network model. The experience module optimizes the experience replay mechanism in the trained deep Q-network model by introducing a priority experience replay mechanism, quantifying the importance of experience samples based on temporal difference error, prioritizing the sampling of high-value samples and introducing importance weight correction, and initializing new samples with the maximum temporal difference error to obtain an optimized training mechanism. The detection module performs beam misalignment detection and adjustment processing on the output of the trained deep Q network model, and establishes a beam misalignment detection and adjustment mechanism. The misalignment criterion is defined based on the half-power beamwidth. When the tracking error exceeds the threshold, a local beam re-measurement process is triggered. The optimal direction is repositioned by combining the surrounding beam signal strength to obtain the adjusted beam direction, which is used to maintain link stability.

[0080] This invention provides a terminal device comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, graphics processing units (GPUs), tensor processing units (TPUs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to achieve a corresponding method flow or function. The processor described in this embodiment can be used for the operation of a high-dynamic scene beam tracking method based on deep reinforcement learning, including: The pilot signal strength received from the communication system is processed to construct a Markov decision process model. A state space is built based on the pilot signal strength, and a discrete action function including beam angle adjustment step size is designed. The ratio of received signal strength is used as the reward function to obtain a dynamic decision process model. A deep Q-network optimization mechanism is then constructed to optimize the dynamic decision process model. This optimization employs a two-layer fully connected neural network to approximate the Q-value function, introduces a target network and a random experience replay mechanism to stabilize the training process, balances exploration and utilization through a greedy strategy, and dynamically adjusts the learning rate to accelerate convergence. The training objective is to minimize the Bellman residual loss function, resulting in the trained deep Q-network model. The network model is optimized by introducing a priority experience replay mechanism. The importance of experience samples is quantified based on temporal difference error. High-value samples are sampled first, and importance weights are introduced for correction. New samples are initialized using the maximum temporal difference error, resulting in an optimized training mechanism. Beam misalignment detection and adjustment are performed on the output of the trained deep Q-network model. A beam misalignment detection and adjustment mechanism is established, defining a misalignment criterion based on half-power beamwidth. When the tracking error exceeds a threshold, a local beam re-measurement process is triggered. The optimal direction is repositioned by combining the surrounding beam signal strength to obtain the adjusted beam direction, which is used to maintain link stability.

[0081] Please see Figure 9 The terminal device is a computer device. In this embodiment, the computer device 60 includes a processor 61, a memory 62, and a computer program 63 stored in the memory 62 and executable on the processor 61. When executed by the processor 61, the computer program 63 implements the high-dynamic scene beam tracking method based on deep reinforcement learning in this embodiment. To avoid repetition, details are omitted here. Alternatively, when executed by the processor 61, the computer program 63 implements the functions of each model / unit in the high-dynamic scene beam tracking system based on deep reinforcement learning in this embodiment. To avoid repetition, details are omitted here.

[0082] Computer device 60 can be a desktop computer, laptop, handheld computer, cloud server, or other computing device. Computer device 60 may include, but is not limited to, a processor 61 and a memory 62. Those skilled in the art will understand that... Figure 9 This is merely an example of computer device 60 and does not constitute a limitation on computer device 60. It may include more or fewer components than shown, or combine certain components, or different components. For example, computer device may also include input / output devices, network access devices, buses, etc.

[0083] The processor 61 may be a Central Processing Unit (CPU), or other general-purpose processors, graphics processing units (GPUs), tensor processing units (TPUs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0084] The memory 62 can be an internal storage unit of the computer device 60, such as a hard disk or RAM of the computer device 60. The memory 62 can also be an external storage device of the computer device 60, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., provided on the computer device 60.

[0085] Furthermore, the memory 62 may include both internal storage units of the computer device 60 and external storage devices. The memory 62 is used to store computer programs and other programs and data required by the computer device. The memory 62 can also be used to temporarily store data that has been output or will be output.

[0086] Please see Figure 10 The terminal device is an electronic device 600, which is manifested in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including storage unit 620 and processing unit 610), a display unit 640, etc.

[0087] The storage unit stores program code, which can be executed by the processing unit 610 to perform the steps described in the method section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 610 can perform actions such as... Figure 2 The steps are shown in the figure.

[0088] Storage unit 620 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 6201 and / or cache memory 6202, and may further include a read-only memory (ROM) 6203.

[0089] Storage unit 620 may also include a program / utility 6204 having a set (at least one) program module 6205, such program module 6205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0090] Bus 630 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the multiple bus structures.

[0091] Electronic device 600 can also communicate with one or more external devices 700 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 600, and / or with any device that enables electronic device 600 to communicate with one or more other computing devices (e.g., router, modem). This communication can be performed via input / output interface 650. Furthermore, electronic device 600 can also communicate with one or more networks (e.g., local area network, wide area network, and / or public network, such as the Internet) via network adapter 660. Network adapter 660 can communicate with other modules of electronic device 600 via bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0092] Example 4 This invention also provides a storage medium, specifically a computer-readable storage medium, which is a memory device in a terminal device for storing programs and data. It is understood that the computer-readable storage medium here can include both built-in storage media in the terminal device and extended storage media supported by the terminal device; it can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor, which can be one or more computer programs (including program code). More specific examples of the computer-readable storage medium include: an electrical connection with one or more wires, a portable disk, a hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical fiber, portable compact disk read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.

[0093] Computer-readable storage media also include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium can also be any readable medium other than a readable storage medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium can be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, radio frequency, etc., or any suitable combination thereof.

[0094] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0095] One or more instructions stored in a computer-readable storage medium can be loaded and executed by a processor to implement the corresponding steps of the high dynamic scene beam tracking method based on deep reinforcement learning in the above embodiments; one or more instructions in the computer-readable storage medium are loaded and executed by the processor in the following steps: The pilot signal strength received from the communication system is processed to construct a Markov decision process model. A state space is built based on the pilot signal strength, and a discrete action function including beam angle adjustment step size is designed. The ratio of received signal strength is used as the reward function to obtain a dynamic decision process model. A deep Q-network optimization mechanism is then constructed to optimize the dynamic decision process model. This optimization employs a two-layer fully connected neural network to approximate the Q-value function, introduces a target network and a random experience replay mechanism to stabilize the training process, balances exploration and utilization through a greedy strategy, and dynamically adjusts the learning rate to accelerate convergence. The training objective is to minimize the Bellman residual loss function, resulting in the trained deep Q-network model. The network model is optimized by introducing a priority experience replay mechanism. The importance of experience samples is quantified based on temporal difference error. High-value samples are sampled first, and importance weights are introduced for correction. New samples are initialized using the maximum temporal difference error, resulting in an optimized training mechanism. Beam misalignment detection and adjustment are performed on the output of the trained deep Q-network model. A beam misalignment detection and adjustment mechanism is established, defining a misalignment criterion based on half-power beamwidth. When the tracking error exceeds a threshold, a local beam re-measurement process is triggered. The optimal direction is repositioned by combining the surrounding beam signal strength to obtain the adjusted beam direction, which is used to maintain link stability.

[0096] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0097] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0098] The beam tracking capability of this invention will now be described in detail. The simulation experiment consists of two parts: real trajectory and statistical error analysis. The statistical error is expressed as RMSE and was obtained through 3000 Monte Carlo repetitions. Table 2 shows the training parameter configuration when using the DQN algorithm.

[0099] Table 2 DQN training parameter settings

[0100] like Figure 6 The figure shows the beam tracking results based on DQN. The blue line represents the actual incident angle, the red dashed line represents the angle predicted by the DQN model, and the black dashed line represents the beam failure boundary. As can be seen from the figure, the predicted AoA generally closely follows the actual AoA, basically falling between the two black boundary lines, indicating that DQN can effectively track dynamically changing angles and avoid beam failure. Although the red prediction line shows slight fluctuations at certain times, it does not cross the beam failure boundary, indicating that the model has good tracking accuracy and robustness. Overall, Figure 6 The effectiveness of the DQN algorithm in angle tracking tasks was verified. Its prediction results can fluctuate stably around the true value and always remain within the allowable beam coverage range.

[0101] Figure 7 The simulation settings changed the motion model from uniform circular motion to variable-speed circular motion. From Figure 7 As can be seen, in dynamic scenarios with trajectory switching points, the DQN algorithm demonstrates its tracking ability and adaptability. The predicted trajectory always closely follows the real AoA before and after the switching point, and the tracking process is smooth and does not exceed the beam failure boundary. Figure 8 The results show the RMSE variations of DQN and EKF under different SNR conditions when the number of antennas is fixed at N=64. EKF performs exceptionally well at high SNR, with the RMSE rapidly decreasing to a minimum. However, the error rises sharply at low SNR, indicating its extreme sensitivity to noise. In contrast, DQN exhibits a more gradual RMSE variation across all SNR conditions, maintaining good tracking performance even at low SNR, demonstrating excellent robustness. Therefore, DQN is more advantageous in complex noisy environments.

[0102] In summary, this invention presents a beam tracking method and system for high-dynamic scenes based on deep reinforcement learning, effectively addressing the limitations of traditional beam tracking methods in high-dynamic scenarios. By constructing a Markov decision process model, the beam tracking problem is transformed into a reinforcement learning task. Pilot signal strength is used to define the state space, action space, and reward function, enabling the system to adapt to environmental changes. The deep Q-network optimization mechanism uses a neural network to approximate the Q-value function, introducing a target network and experience replay for stable training. A greedy strategy balances exploration and utilization, accelerating convergence. The priority experience replay mechanism prioritizes sampling based on temporal differential errors, improving sample utilization efficiency and reducing training bias. The beam misalignment detection mechanism sets a threshold based on half-power beamwidth, triggering a re-initialization process to ensure link continuity. Simulation results show that this method can accurately track beam angles in high-speed moving scenarios, avoiding misalignment, and maintains good performance in low signal-to-noise ratio environments. Compared to traditional methods such as EKF, it exhibits higher robustness, real-time performance, and adaptability. Overall, this invention improves the accuracy and reliability of beam tracking, is suitable for high-speed dynamic communication scenarios such as UAV swarms, and provides an effective solution for future wireless communication systems.

[0103] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0104] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0105] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0106] In the embodiments provided by this invention, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0107] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0108] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0109] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random-access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0110] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0111] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0112] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0113] The above content is only for illustrating the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of the claims of this invention.

Claims

1. A high dynamic scene beam tracking method based on deep reinforcement learning, characterized in that, The method comprises the following steps: S1, processing the pilot signal strength received from the communication system, constructing a Markov decision process model, constructing a state space based on the pilot signal strength, designing a discrete action function containing a beam angle adjustment step, and taking a received signal strength ratio as a reward function to obtain a dynamic decision process model; S2, building a deep Q network optimization mechanism, performing deep Q network optimization processing on the dynamic decision process model, wherein a two-layer fully connected neural network is used to approximate the Q value function, a target network and a random experience replay mechanism are introduced to stabilize the training process, a greedy strategy is used to balance exploration and utilization, and the learning rate is dynamically adjusted to speed up convergence, and the Bellman residual loss function is minimized as the training target to obtain the trained deep Q network model; S3, optimizing the experience replay mechanism in the trained deep Q network model, introducing a priority experience replay mechanism, quantifying the importance of experience samples based on the timing difference error, preferentially sampling high-value samples and introducing importance weight correction, and initializing new samples using the maximum timing difference error to obtain an optimized training mechanism; S4, performing beam misalignment detection and adjustment processing on the output of the trained deep Q network model, establishing a beam misalignment detection and adjustment mechanism, wherein a misalignment criterion based on a half-power beam width is defined, and when the tracking error exceeds a threshold, a local beam resurvey process is triggered, the optimal direction is repositioned combined with the signal strength of the surrounding beams, and the adjusted beam direction is obtained to maintain link stability.

2. The deep reinforcement learning based high dynamic scene beam tracking method according to claim 1, characterized in that, In step S1, the state space includes the received signals of the current time slot and the previous time slot, and the state space is all the states accessible by the agent in the interaction process; the action space is designed to be uniformly distributed around the maximum angle change range, and the action space A is represented as: wherein, is the maximum angle change, is the action step size, is the number of actions, is the action, is the action index; The reward function is constructed based on the trend of the received signal strength, and the reward function is represented as: wherein, and are lower and upper threshold values, respectively, is the number of pilots before the action in each time slot, is the number of pilot symbols in each time slot, is the received signal at the pilot symbol in the time slot, is the upper threshold value, is the lower threshold value.

3. The deep reinforcement learning based high dynamic scene beam tracking method according to claim 2, characterized in that, the maximum angle change According to the maximum speed of the mobile terminal and the minimum distance calculation, expressed as: wherein, is the maximum speed of the mobile terminal, is the time slot duration, is the minimum distance; the action step is represented as: wherein is the output layer.

4. The deep reinforcement learning based high dynamic scene beam tracking method according to claim 1, characterized in that, In step S2, the deep Q-network includes an input layer, a hidden layer, and an output layer, wherein the input layer includes the layer before the current time slot. Each pilot symbol signal and the previous time slot There are 1 signal, and the hidden layer includes a first hidden layer and a second hidden layer. The network size of the first hidden layer is 1. The size of the second hidden layer network is The output layer outputs the Q-values ​​of all possible actions; the target network has the same structure as the main network, and the weights of the main network are copied to the target network every C training iterations.

5. The deep reinforcement learning based high dynamic scene beam tracking method according to claim 4, characterized in that, The dynamic adjustment of the greedy strategy is based on the current round number and the total round number, the exploration rate is represented as: wherein is the current round number, is the total round number.

6. The deep reinforcement learning based high dynamic scene beam tracking method according to claim 1, characterized in that, In step S3, in the priority experience replay mechanism, the sampling probability of the experience sample is determined by the priority determined by the timing difference error and the hyperparameter, and is represented as: wherein, is an empirical sample under control of hyperparameters priority of is a sample under control of hyperparameters priority; The priority Based on timing difference error The calculation is expressed as: wherein is a constant and greater than 0.

7. The deep reinforcement learning based high dynamic scene beam tracking method according to claim 6, characterized in that, The timing difference error Is: wherein, to reward, is a discount factor, is a current state-action value, is a next state-action value of the target network output.

8. The deep reinforcement learning based high dynamic scene beam tracking method according to claim 1, characterized in that, In step S4, the misalignment criterion is based on the root mean square error of beam tracking, and is defined as half of the half-power beam width, and is represented as: wherein, represents the half-power beamwidth, which is approximated as: ; is the number of receive antennas; and the beam re-initialization procedure is triggered when the tracking error exceeds .

9. The deep reinforcement learning based high dynamic scene beam tracking method according to claim 1, characterized in that, The beam reinitialization process uses a compressed sensing spectrum estimation method to reposition the incoming direction, and resets the experience pool and network weights of the deep Q network to start a new round of tracking iteration.

10. A high dynamic scene beam tracking system based on deep reinforcement learning, characterized in that, It comprises: a state module for processing the pilot signal strength received from the communication system, constructing a Markov decision process model, constructing a state space based on the pilot signal strength, designing a discrete action function containing a beam angle adjustment step, and taking a received signal strength ratio as a reward function to obtain a dynamic decision process model; an optimization module for building a deep Q network optimization mechanism, performing deep Q network optimization processing on the dynamic decision process model, wherein a two-layer fully connected neural network is used to approximate the Q value function, a target network and a random experience replay mechanism are introduced to stabilize the training process, a greedy strategy is used to balance exploration and utilization, and the learning rate is dynamically adjusted to speed up convergence, and the Bellman residual loss function is minimized as the training target to obtain the trained deep Q network model; An experience module optimizes an experience replay mechanism in the trained deep Q network model, introduces a priority experience replay mechanism, quantifies experience sample importance based on a time difference error, preferentially samples high-value samples and introduces importance weight correction, initializes new samples using maximum time difference error, and obtains an optimized training mechanism. A detection module performs beam misalignment detection and adjustment processing on the output of the trained deep Q network model, establishes a beam misalignment detection and adjustment mechanism, defines a misalignment criterion based on a half-power beam width, triggers a local beam re-measurement process when the tracking error exceeds a threshold, repositions the optimal direction combined with the signal strength of the surrounding beams, obtains an adjusted beam direction, and maintains link stability.

Images