An obstacle avoidance and tracking method based on deep reinforcement learning

Abstract

This invention presents an obstacle avoidance and tracking method based on deep reinforcement learning, belonging to the field of path planning. Specifically, it is a multi-UAV reconnaissance near-end strategy optimization method based on a distributed reinforcement learning framework. The method achieves precise avoidance of dynamic obstacles and efficient tracking of abnormal radiation sources through the deep fusion of a pressure perception mechanism and an improved SAC reinforcement learning method. The pressure perception module quantifies environmental pressure into numerical indicators and dynamically adjusts the conservatism of the decision-making strategy accordingly. Based on this, a dual-Q network architecture of the improved SAC algorithm is used for decision optimization, employing both conservative and aggressive strategies for parallel evaluation, thus improving the robustness of the strategy while ensuring exploration efficiency. This invention operates in a continuous action space, ensuring the smoothness and stability of the UAV's trajectory. This invention enables UAVs to intelligently track abnormal radiation sources under high-pressure environments in complex urban electromagnetic spectrum operations.

Description

Technical Field

[0001] This invention relates to an obstacle avoidance and tracking method based on reinforcement learning, belonging to the field of path planning. Technical Background

[0002] With the rapid development of modern cities, the urban electromagnetic environment has become unprecedentedly complex. Electromagnetic radiation from various communication devices, industrial equipment, and civilian electronic products intertwine, forming a dense and complex electromagnetic spectrum environment. The widespread adoption of drone technology has further exacerbated this situation: on the one hand, the extensive use of commercial and consumer drones has generated a large number of new electromagnetic signal sources. On the other hand, the demand for drones to detect and track abnormal radiation sources is increasing daily in law enforcement, security, and other fields. In security operations for major events, rapidly tracking suspicious electromagnetic signal sources has become a critical task. However, because our monitoring equipment and ordinary, legitimate drones also operate in the same electromagnetic space, signal interference is highly likely. This "electromagnetic collateral damage" not only affects mission effectiveness but may also lead to the failure of legitimate equipment. Therefore, in the complex urban electromagnetic environment, building an intelligent obstacle avoidance and tracking system has become one of the core tasks of electromagnetic spectrum warfare.

[0003] Currently, traditional methods for UAV path planning in urban environments mainly fall into three categories: sampling-based, search-based, and optimization-based methods. Sampling-based planning (such as RRT and PRM) is highly efficient in handling high-dimensional spaces and has probabilistic completeness, but the paths are not smooth enough and it is difficult to guarantee optimality. Search-based planning (such as A* and D*) has good theoretical completeness and can guarantee the optimal path, but it suffers from the curse of dimensionality and has high storage overhead. Optimization-based planning (such as MPC) can generate smooth trajectories and is easy to constrain, but it is prone to getting trapped in local optima and has high computational complexity. Reinforcement learning algorithms, through end-to-end learning and continuous exploration and optimization, can adapt to complex dynamic environments, integrate multi-source sensing information, and improve computational efficiency with experience accumulation. Especially in complex urban electromagnetic environments, they can learn signal propagation characteristics and environmental interaction patterns, balance exploration and utilization to avoid local optima, dynamically adjust decision-making strategies to improve robustness, and reduce online computational burden to improve real-time performance, effectively overcoming the limitations of traditional methods.

[0004] However, traditional reinforcement learning suffers from drawbacks such as low sample efficiency, unstable training, and poor policy generalization ability. Furthermore, it is inefficient in exploring high-dimensional action spaces, and its reward function design struggles to balance the demands of multiple objectives, making it challenging to ensure real-time performance and accuracy. In real-world UAV tracking missions, the environmental state space is vast, and traditional reinforcement learning struggles to adapt quickly to unexpected situations. Therefore, designing resilient reinforcement learning algorithms to enhance UAV path planning capabilities under high-pressure environments, achieve efficient strikes against anomalous radiation sources, and balance exploration efficiency with decision stability are pressing issues that need to be addressed. Summary of the Invention

[0005] To address the challenge of stably tracking anomalous radiation sources under high-pressure environments, this invention provides a deep reinforcement learning-based obstacle avoidance and tracking method. By deeply integrating a pressure perception mechanism with an improved SAC (Soft Actor-Critic) reinforcement learning method, it achieves precise avoidance of dynamic obstacles and efficient tracking of anomalous radiation sources. The pressure perception module quantifies environmental pressure into numerical indicators and dynamically adjusts the conservatism of the decision-making strategy accordingly. Based on this, a dual-Q network architecture of the improved SAC algorithm is used for decision optimization, employing both conservative and aggressive strategies for parallel evaluation to improve the robustness of the strategy while ensuring exploration efficiency. This invention operates in a continuous action space, ensuring the smoothness and stability of the UAV's trajectory. This invention effectively solves the challenge of intelligently tracking anomalous radiation sources under high-pressure environments in complex urban electromagnetic spectrum warfare, achieving anomalous radiation source tracking based on elastic reinforcement learning.

[0006] The objective of this invention is achieved through the following technical solution.

[0007] This invention discloses an obstacle avoidance and tracking method based on deep reinforcement learning, which is a multi-UAV reconnaissance near-end strategy optimization method based on a distributed reinforcement learning framework, comprising the following steps:

[0008] Step 1: Construct a multi-layered environmental sensing system to acquire environmental state information. The multi-layered environmental sensing system includes a terrain layer, an electromagnetic layer, and a pressure layer.

[0009] Step 2: Construct a hierarchical, resilient obstacle avoidance decision-making framework, comprising a stress perception module, a value assessment module, and a strategy decision-making module. The stress perception module performs risk assessment based on the comprehensive environmental stress output by the stress layer, the value assessment evaluates the state-action value under the current stress state, and the strategy network generates decision instructions adapted to the current stress based on the value assessment. The three modules form a closed-loop feedback mechanism.

[0010] Step 3: Design a dynamic reward mechanism. Design tracking rewards based on environmental status information, design risk avoidance rewards based on stress level assessment, and dynamically adjust the reward weights according to the stress level.

[0011] Step 4: Based on the hierarchical elastic obstacle avoidance decision framework built in Step 2, and using the dynamic reward mechanism designed in Step 3, construct the optimal obstacle avoidance and tracking strategy under different pressure environments, and realize obstacle avoidance and tracking of the target UAV according to the optimal obstacle avoidance and tracking strategy.

[0012] Furthermore, the method for implementing step one is as follows:

[0013] A unified two-dimensional coordinate system is established, and the coordinate system space is discretized into a grid to obtain a grid map.

[0014] The multi-layer environmental perception system is divided into three main layers: the terrain layer, the electromagnetic layer, and the pressure layer.

[0015] The terrain layer primarily contains information on non-radioactive obstacles within the city. Each obstacle... i Represented as:

[0016] o i ={p i ,s i ,b i ,r i} (1)

[0017] Where p i Let s be the coordinates of the obstacle's center. i For geometric dimensions, b i r is the electromagnetic wave blocking coefficient. i The risk factor is determined by mapping all obstacle information onto a grid map, forming obstacle distribution matrix B.

[0018] B = {b} ij |i,j∈N} (2)

[0019] Where b ij This represents the obstacle distribution information in the i-th row and j-th column of the grid map, where N represents the dimensional range of the grid map.

[0020] The electromagnetic layer contains k UAVs, and its electromagnetic radiation intensity model is as follows:

[0021]

[0022] Where P k For radiated power, (x k ,y k ) represents the location of the drone, σ k This refers to the radiation range parameter.

[0023] The pressure layer introduces a pressure metric P to quantify the threat level of the current environment:

[0024] P=α1·D t +β1·E i +γ1·R o (4)

[0025] Where D t For the distance pressure from the target, E i Due to environmental disturbance pressure, R o The obstacle risk pressure is represented by α1, β1, and γ1, which are the corresponding weighting coefficients.

[0026] The overall environmental state distribution considering obstacle blocking effects and pressure influences is as follows:

[0027]

[0028] The occlusion coefficient μ(x,y,z) is determined by the obstacle distribution matrix B:

[0029] μ(x,y)=f(B,x,y) (6)

[0030] The pressure regulation function ψ(P) is:

[0031]

[0032] Where P0 is the pressure threshold and λ is the pressure regulation sensitivity coefficient.

[0033] Furthermore, the state obtained in step one is represented as follows:

[0034] s t ={B,E,P} (8)

[0035] Furthermore, the pressure sensing module described in step two is obtained based on the pressure index P of the pressure layer in step one.

[0036] The pressure assessment function of the pressure sensing module is expressed as:

[0037]

[0038] Where τ is the time variable used for pressure integral calculation. w1, w2, and w3 are weighting coefficients, corresponding to the pressure value, pressure change rate, and pressure cumulative effect, respectively. V p This is a stress assessment value used for a comprehensive assessment of environmental stress. Based on the assessment value V... p The stress state is dynamically scored.

[0039] Furthermore, the value assessment module described in step two is designed with a dynamically balanced dual-Q network structure, including a conservative value network Q. c and radical value network Q a :

[0040] When the environmental pressure value V p >0.5:

[0041] Q c (s t ,a t ;θ c )=E[r t (1+α2V p )+γQ c (s t+1 ,a t+1;θ c′ (10)

[0042] When the environmental pressure value V p When ≤0.5:

[0043] Q a (s t ,a t ;θ a )=E[r t (1-β2V p )+γQ a (s t+1 ,a t+1 ;θ a′ (11)

[0044] α2 and β2 are pressure influence coefficients, which are adjusted by the immediate reward r. t This allows conservative networks to gain greater weight under high pressure, while radical networks gain greater weight under low pressure.

[0045] When V p When the loss is greater than 0.5, the loss function is defined as:

[0046]

[0047] When V p When ≤0.5, the loss function is defined as:

[0048]

[0049] Where the target value y a y c Calculated in the following way:

[0050] y c =r t (1+α2V p )+γ·Q c (s t+1 ,a t+1 ;θ c′ (14)

[0051] y a =r t (1-β2V p )+γ·Q a (s t+1 ,a t+1 ;θ a′ (15)

[0052] Where γ is the discount factor and α is the entropy regularization coefficient.

[0053] Furthermore, the action decision module described in step two dynamically selects the strategy network output based on the environmental pressure value:

[0054] When the environmental pressure value V p When the value is greater than 0.5, a conservative strategy network is adopted:

[0055]

[0056] Where π c This represents a conservative strategy network. φ c This represents the network parameters of a conservative policy. τ is the temperature parameter, controlling the degree of exploration by the policy. a′ represents all possible actions in the action space.

[0057] When the environmental pressure value V p When the value is ≤0.5, an aggressive strategy network is adopted:

[0058]

[0059] Where π a Describes an aggressive policy network, φ a This represents the network parameters for an aggressive strategy.

[0060] The loss function of the policy network in V p When >0.5, it is defined as:

[0061]

[0062] In V p When ≤0.5:

[0063]

[0064] Where ρ represents the state distribution. This represents the expectation under the state distribution ρ. α is the entropy regularization coefficient, used to balance exploration and exploitation.

[0065] Furthermore, the method for implementing step three is as follows:

[0066] The basic tracking reward, considering target distance and approach speed, is expressed as:

[0067]

[0068] Where d t This represents the current distance to the target, η is the speed bonus coefficient, and Δt is the time interval.

[0069] The obstacle avoidance safety reward is a penalty imposed on the distance to the obstacle through an exponential function. This reward is expressed as:

[0070]

[0071] Where k s d represents the obstacle avoidance sensitivity coefficient. obs d represents the distance to the obstacle. safe This indicates the safe distance threshold.

[0072] Simultaneously, a stress regulation reward r is introduced. press =-P·|a t | 2 and stability reward r stab =-|a t -a t-1 | 2 These constraints address the system's behavioral amplitude and action continuity under high-pressure conditions.

[0073] The final comprehensive reward function achieves adaptive fusion of various rewards through dynamic weighting coefficients:

[0074] r t =κ1·r track +κ2·r safe +κ3·r press +κ4·r stab (twenty two)

[0075] Furthermore, the implementation method for step four is as follows:

[0076] Step 4.1: Initialize the conservative value network parameters θ c Radical value network parameter θ a Conservative strategy network parameters φ c Radical strategy network parameters φ a Initial environmental conditions s0 and pressure value V p =0.

[0077] Step 4.2: At the start of each training round, reset the environment state s. t and pressure value V p =0.

[0078] Step 4.3: Based on the current environmental state s t and pressure value V p Actions are sampled from the corresponding policy network.

[0079] Step 4.4: Perform action a t Next environmental state s t+1 And receive the reward r set in step three. t .

[0080] Step 4.5: Update the parameters of the value network and policy network based on the status and reward in Step 4.4.

[0081] Step 4.6: Once the target state is reached or the maximum number of steps is exceeded, the current round ends.

[0082] Step 4.7: Reset the environmental state and stress value to set the environmental state to s0, V p =0. Repeat training steps 4.3 to 4.6 until convergence. Output network parameters θ. c θ a φ c φ a .

[0083] Step 4.8: The training process from Step 4.2 to Step 4.7 is carried out in an end-to-end manner. By continuously interacting with the environment, experience is accumulated and the parameters of each network are updated, and finally a safe and efficient obstacle avoidance and tracking strategy that can be implemented in complex electromagnetic environments is obtained. The target UAV is tracked and tracked based on the optimal obstacle avoidance and tracking strategy.

[0084] Beneficial effects:

[0085] 1. This invention discloses an obstacle avoidance and tracking method based on deep reinforcement learning. By constructing a multi-level environmental perception system, it decomposes complex adversarial environments into three dimensions: an obstacle layer, an electromagnetic layer, and a pressure layer, achieving comprehensive perception and quantitative representation of environmental threats. In particular, the introduction of a pressure metric P effectively integrates distance pressure, environmental interference pressure, and obstacle risk pressure, enabling the system to accurately assess the current situation and provide a reliable basis for subsequent decision-making.

[0086] 2. This invention discloses an obstacle avoidance and tracking method based on deep reinforcement learning. The designed hierarchical elastic obstacle avoidance decision-making framework features strong adaptability and robustness. A dual-Q network structure enables dynamic switching between conservative and aggressive evaluations, and a pressure adjustment factor is used to adaptively adjust strategy selection, allowing the UAV to flexibly choose avoidance strategies based on environmental pressure. Simultaneously, based on an improved SAC algorithm, a dynamic balance between exploration and exploitation is achieved through adaptive temperature parameters, improving the reliability of the decision.

[0087] 3. This invention discloses a deep reinforcement learning-based obstacle avoidance and tracking method. The dynamic reward mechanism comprehensively considers four aspects: tracking performance, obstacle avoidance safety, stress response, and control stability. The weight coefficients of each reward component can be dynamically adjusted according to the stress state. This invention ensures the safety of the UAV while improving tracking efficiency, and achieves rapid response to environmental stress while maintaining control stability, significantly improving the overall performance of the UAV in complex adversarial environments.

[0088] 4. The obstacle avoidance and tracking method disclosed in this invention, based on deep reinforcement learning, achieves the above three beneficial effects by adopting a UAV reinforcement learning method based on a hierarchical elastic framework. Through the deep integration of pressure perception mechanism and improved SAC (Soft Actor-Critic) reinforcement learning method, it can effectively solve the challenge of UAVs intelligently tracking abnormal radiation sources in high-pressure environments in complex urban electromagnetic spectrum operations, and realize abnormal radiation source tracking based on elastic reinforcement learning. Attached Figure Description

[0089] Figure 1 This is a schematic diagram of terrain sensing.

[0090] Figure 2 This is a schematic diagram of electromagnetic sensing.

[0091] Figure 3 This is a flowchart of the reinforcement learning obstacle avoidance and tracking algorithm disclosed in this invention. Detailed Implementation

[0092] To enable those skilled in the art to more deeply understand the implementation ideas of the present invention, the technical solutions of the present invention will be carefully and clearly described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other implementation cases obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention. The specific steps of the embodiments of the present invention will be described below in conjunction with specific scenarios.

[0093] Example:

[0094] The abnormal radiation source tracking scenario in this embodiment is as follows: Figure 1 As shown, a multi-layered environmental perception system for multi-agent collaboration was constructed over a range of 10km × 10km, deploying 3 red-team jammers, 2 white-team UAVs, and 1 blue-team UAV. During each mission evolution, the system records key data. This invention considers collaborative localization of anomalous radiation sources based on multiple agents.

[0095] This embodiment discloses an obstacle avoidance and tracking method based on deep reinforcement learning, and the specific implementation steps are as follows:

[0096] Step 1: Construct a multi-layered environmental perception system, including a terrain layer, an electromagnetic layer, and a pressure layer, to acquire environmental status information.

[0097] First, a unified two-dimensional coordinate system is established, and then the coordinate system space is discretized into a grid to obtain a grid map.

[0098] The multi-layer environmental perception system is divided into three main layers: the terrain layer, the electromagnetic layer, and the pressure layer.

[0099] like Figure 1 As shown, the terrain layer primarily contains information on non-radioactive obstacles within the city. Each obstacle... i Represented as:

[0100] o i ={p i ,s i ,b i ,r i} (twenty three)

[0101] Where p i Let s be the coordinates of the obstacle's center. i For geometric dimensions, b i r is the electromagnetic wave blocking coefficient. i The risk factor is determined by mapping all obstacle information onto a grid map, forming obstacle distribution matrix B.

[0102] B = {b} ij |i,j∈N} (24)

[0103] Where b ij This represents the obstacle distribution information in the i-th row and j-th column of the grid map, where N represents the dimensional range of the grid map.

[0104] like Figure 2 As shown, the electromagnetic layer contains 6 drones, and the electromagnetic radiation intensity model of the k-th drone is as follows:

[0105]

[0106] Where P k For radiated power, (x k ,y k ) represents the location of the drone, σ k This refers to the radiation range parameter.

[0107] The pressure layer introduces a pressure metric P to quantify the threat level of the current environment:

[0108] P=α1·D t +β1·E i +γ1·R o (26)

[0109] Where D t For the distance pressure from the target, E i Due to environmental disturbance pressure, R o The obstacle risk pressure is represented by α1, β1, and γ1, which are the corresponding weighting coefficients, set to 0.4, 0.3, and 0.3, respectively.

[0110] The overall environmental state distribution considering obstacle blocking effects and pressure influences is as follows:

[0111]

[0112] The occlusion coefficient μ(x,y,z) is determined by the obstacle distribution matrix B:

[0113] μ(x,y)=f(B,x,y) (28)

[0114] The pressure regulation function ψ(P) is:

[0115]

[0116] Where P0 is the pressure threshold and λ is the pressure regulation sensitivity coefficient, set to 0 and 1 respectively.

[0117] The overall state is represented as follows:

[0118] s t ={B,E,P} (30)

[0119] Step 2: Construct a hierarchical, resilient obstacle avoidance decision-making framework, comprising a stress perception module, a value assessment module, and a strategy decision-making module. The stress perception module performs risk assessment based on the comprehensive environmental stress output by the stress layer, the value assessment evaluates the state-action value under the current stress state, and the strategy network generates decision instructions adapted to the current stress based on the value assessment. The three modules form a closed-loop feedback mechanism.

[0120] The pressure assessment function of the pressure sensing module is expressed as:

[0121]

[0122] Where τ is the time variable used for pressure integral calculation. w1, w2, and w3 are weighting coefficients, corresponding to the pressure value, pressure change rate, and pressure cumulative effect, respectively, and are set to 0.4, 0.3, and 0.3. V p This is a stress assessment value used for a comprehensive assessment of environmental stress. Based on the assessment value V... p The stress state is dynamically scored.

[0123] The value assessment module is designed with a dynamically balanced dual-Q network structure, including a conservative value network Q. c and radical value network Q a :

[0124] When the environmental pressure value V p >0.5:

[0125] Q c (s t ,a t ;θ c)=E[r t (1+α2V p )+γQ c (s t+1 ,a t+1 ;θ c′ (32)

[0126] When the environmental pressure value V p When ≤0.5:

[0127] Q a (s t ,a t ;θ a )=E[r t (1-β2V p )+γQ a (s t+1 ,a t+1 ;θ a′ (33)

[0128] α2 and β2 are pressure influence coefficients, both set to 1. The immediate reward r is adjusted... t This allows conservative networks to gain greater weight under high pressure, while radical networks gain greater weight under low pressure.

[0129] When V p When the loss is greater than 0.5, the loss function is defined as:

[0130]

[0131] When V p When ≤0.5, the loss function is defined as:

[0132]

[0133] Where the target value y a y c Calculated in the following way:

[0134] y c =r t (1+α2V p )+γ·Q c (s t+1 ,a t+1 ;θ c′ (36)

[0135] y a =r t (1-β2V p )+γ·Q a (s t+1 ,a t+1 ;θ a′ (37)

[0136] Where γ is the discount factor, set to 0.8, and α is the entropy regularization coefficient, set to 0.1.

[0137] The strategy decision-making module dynamically selects the strategy network output based on the environmental stress value:

[0138] When the environmental pressure value V p When the value is greater than 0.5, a conservative strategy network is adopted:

[0139]

[0140] Where π c This represents a conservative strategy network. φ c This represents the network parameters for a conservative policy. τ is the temperature parameter, controlling the degree of exploration by the policy, and is set to 0.5. a′ represents all possible actions in the action space.

[0141] When the environmental pressure value V p When the value is ≤0.5, an aggressive strategy network is adopted:

[0142]

[0143] Where π a Describes an aggressive policy network, φ a This represents the network parameters for an aggressive strategy.

[0144] The loss function of the policy network in V p When >0.5, it is defined as:

[0145]

[0146] In V p When ≤0.5:

[0147]

[0148] Where ρ represents the state distribution. This represents the expectation under the state distribution ρ. α is the entropy regularization coefficient, set to 0.8, used to balance exploration and exploitation.

[0149] Step 3: Design a dynamic reward mechanism. Design tracking rewards based on environmental status information, design risk avoidance rewards based on stress level assessment, and dynamically adjust the reward weights according to the stress level.

[0150] The basic tracking reward, considering target distance and approach speed, is expressed as:

[0151]

[0152] Where d tThis represents the current distance to the target, η is the speed bonus coefficient, set to 1, and Δt is the time interval.

[0153] The obstacle avoidance safety reward is a penalty imposed on the distance to the obstacle through an exponential function. This reward is expressed as:

[0154]

[0155] Where k s This represents the obstacle avoidance sensitivity coefficient, set to 1, d obs d represents the distance to the obstacle. safe This indicates the safe distance threshold, set to 20 meters.

[0156] Simultaneously, a stress regulation reward r is introduced. press =-P·|a t | 2 and stability reward r stab =-|a t -a t-1 | 2 These constraints address the system's behavioral amplitude and action continuity under high-pressure conditions.

[0157] The final comprehensive reward function achieves adaptive fusion of various rewards through dynamic weighting coefficients:

[0158] r t =κ1·r track +κ2·r safe +κ3·r press +κ4·r stab (44)

[0159] Among them, κ1, κ2, κ3, and κ4 are all set to 0.25.

[0160] Step 4: Execute the algorithm training and testing process to achieve the optimal obstacle avoidance and tracking strategy under different stress environments.

[0161] Step 4.1: Initialize the conservative value network parameters θ c Radical value network parameter θ a Conservative strategy network parameters φ c Radical strategy network parameters φ a Initial environmental conditions s0 and pressure value V p =0.

[0162] Step 4.2: At the start of each training round, reset the environment state s. t and pressure value V p =0.

[0163] Step 4.3: Based on the current environmental state s t and pressure value Vp Actions are sampled from the corresponding policy network.

[0164] Step 4.4: Perform action a t Next environmental state s t+1 And receive the reward r set in step three. t .

[0165] Step 4.5: Update the parameters of the value network and policy network based on the status and reward in Step 4.4.

[0166] Step 4.6: Once the target state is reached or the maximum number of steps is exceeded, the current round ends.

[0167] Step 4.7: Reset the environmental state and stress value to set the environmental state to s0, V p =0. Repeat training steps 4.3 to 4.6 until convergence. Output network parameters θ. c θ a φ c φ a .

[0168] Step 4.8: The training process from Step 4.2 to Step 4.7 is carried out in an end-to-end manner. By continuously interacting with the environment, experience is accumulated and the parameters of each network are updated, and finally a safe and efficient obstacle avoidance and tracking strategy that can be implemented in complex electromagnetic environments is obtained. The target UAV is tracked and tracked based on the optimal obstacle avoidance and tracking strategy.

[0169] Step 5: During the testing phase, perform the tests according to the following procedure:

[0170] Step 5.1: Load the network parameters θ trained in Step 4 c θ a φ c φ a .

[0171] Step 5.2: Obtain the current environment state s t and pressure value V p .

[0172] Step 5.3: Based on the pressure value V p Select the corresponding policy network and determine the action to be performed.

[0173] Step 5.4: Execute the action and obtain the new state s t+1 and pressure value V p .

[0174] Step 5.5: Repeat steps 5.2 to 5.4. If the target is caught, the test is successful; otherwise, the test fails and retraining is required in a new environment.

[0175] The above detailed description further illustrates the purpose, technical solution, and beneficial effects of the invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An obstacle avoidance and tracking method based on deep reinforcement learning, characterized in that: Includes the following steps, Step 1: Construct a multi-layer environmental sensing system to acquire environmental state information; the multi-layer environmental sensing system includes a terrain layer, an electromagnetic layer, and a pressure layer; The implementation method for step one is as follows: A unified two-dimensional coordinate system is established, and the coordinate system space is discretized into a grid to obtain a grid map; The multi-layer environmental perception system is divided into three main layers: the terrain layer, the electromagnetic layer, and the pressure layer. The terrain layer contains information on radiation-free obstacles in the city; each obstacle... Represented as: （1） in The coordinates of the obstacle's center are... For geometric dimensions, This is the electromagnetic wave blocking coefficient. Risk coefficient; all obstacle information is mapped onto a grid map to form obstacle distribution matrix B: （2） Where b ij This represents the obstacle distribution information in the i-th row and j-th column of the grid map, where N represents the dimensional range of the grid map; The electromagnetic layer contains k UAVs, and its electromagnetic radiation intensity model is as follows: （3） in For radiated power, Location of the drone. For radiation range parameters; The pressure layer introduces a pressure metric P to quantify the threat level of the current environment: （4） in To the pressure of distance from the target, Due to environmental disturbance pressure, For obstacle risk pressure, , , These are the corresponding weighting coefficients; The overall environmental state distribution considering obstacle blocking effects and pressure influences is as follows: （5） Among them, the occlusion coefficient Determined by obstacle distribution matrix B: （6） Pressure regulation function for: （7） Where P0 is the pressure threshold and λ is the pressure regulation sensitivity coefficient; Step 2: Construct a hierarchical, resilient obstacle avoidance decision-making framework, which includes a stress perception module, a value assessment module, and a strategy decision-making module. The stress perception module performs risk assessment based on the comprehensive environmental stress output by the stress layer. The value assessment module is used to assess the state-action value under the current stress state. The strategy network generates decision instructions that adapt to the current stress based on the value assessment. The three modules form a closed-loop feedback mechanism. Step 3: Design a dynamic reward mechanism, design tracking rewards based on environmental status information, design risk avoidance rewards based on stress level assessment, and dynamically adjust the reward weights according to the stress level; Step 4: Based on the hierarchical elastic obstacle avoidance decision framework built in Step 2, and using the dynamic reward mechanism designed in Step 3, construct the optimal obstacle avoidance and tracking strategy under different pressure environments, and realize obstacle avoidance and tracking of the target UAV according to the optimal obstacle avoidance and tracking strategy.

2. The obstacle avoidance and tracking method based on deep reinforcement learning as described in claim 1, characterized in that: The environmental state information obtained in step one is represented as follows: （8）。 3. The obstacle avoidance and tracking method based on deep reinforcement learning as described in claim 1, characterized in that: The pressure sensing module described in step two is based on the pressure index P of the pressure layer in step one. The pressure assessment function of the pressure sensing module is expressed as: （9） Where τ is a time variable used for pressure integral calculation; , , These are weighting coefficients, corresponding to the pressure value, the rate of change of pressure, and the cumulative effect of pressure, respectively; V p This is a pressure assessment value used for a comprehensive assessment of environmental pressure; based on the assessment value V... p The stress state is dynamically scored.

4. The obstacle avoidance and tracking method based on deep reinforcement learning as described in claim 3, characterized in that: The value assessment module described in step two is designed with a dynamically balanced dual-Q network structure, which includes a conservative value network. and radical value networks : When the pressure assessment value V p > 0.5 hours: （10） When the pressure assessment value V p When ≤ 0.5: （11） α2 and β2 are pressure influence coefficients, which are adjusted by the immediate reward r. t This allows conservative networks to gain greater weight under high pressure, while radical networks gain greater weight under low pressure. When V p When the loss is greater than 0.5, the loss function is defined as: （12） When V p When ≤ 0.5, the loss function is defined as: （13） The target value y is calculated in the following way: （14） （15） Where γ is the discount factor and α is the entropy regularization coefficient.

5. The obstacle avoidance and tracking method based on deep reinforcement learning as described in claim 1, characterized in that: The strategy decision-making module described in step two dynamically selects the strategy network output based on the stress assessment value: When the environmental pressure value V p When the value is greater than 0.5, a conservative strategy network is adopted: （16） in Represents a conservative strategy network; Indicates the network parameters of the conservative strategy; The degree of exploration of the control strategy is related to the temperature parameter. Represents all possible actions in the action space; When the environmental pressure value V p When the value is ≤ 0.5, an aggressive strategy network is adopted: （17） in Represents an aggressive policy network. Indicates the network parameters of the aggressive policy; The loss function of the policy network in V p When > 0.5, it is defined as: （18） In V p When ≤ 0.5: （19） in Represents the state distribution; Indicates the state distribution The following expectations; This is the entropy regularization coefficient, used to balance exploration and exploitation.

6. The obstacle avoidance and tracking method based on deep reinforcement learning as described in claim 4, characterized in that: The method for implementing step three is as follows: The basic tracking reward, considering target distance and approach speed, is expressed as: （20） in Indicates the current distance from the target. For speed reward coefficient, For time intervals; The obstacle avoidance safety reward is a penalty imposed on the distance to the obstacle through an exponential function. This reward is expressed as: （21） in This represents the obstacle avoidance sensitivity coefficient. Indicates the distance to the obstacle. Indicates the safe distance threshold; Simultaneously introduce stress regulation rewards and stability rewards These constraints address the system's behavioral amplitude and action continuity under high-pressure conditions. The final comprehensive reward function achieves adaptive fusion of various rewards through dynamic weighting coefficients: （22）。 7. The obstacle avoidance and tracking method based on deep reinforcement learning as described in claim 1, characterized in that: Step four is implemented as follows: Step 4.1: Initialize the conservative value network parameters Radical value network parameters Conservative strategy network parameters Radical strategy network parameters Initial environment state and pressure value V p =0; Step 4.2: At the start of each training round, reset the environment state. and pressure value V p =0; Step 4.3: Based on the current environmental status and pressure value V p Actions are sampled from the corresponding policy network; Step 4.4: Perform the action Next environmental state ; And receive the rewards set in step three. ; Step 4.5: Update the parameters of the value network and policy network based on the status and reward in Step 4.4; Step 4.6: Once the target state is reached or the maximum number of steps is exceeded, the current round ends; Step 4.7: Reset the environmental state and stress value to make the environmental state as follows: V p =0; Repeat training steps 4.3 to 4.6 until convergence; Output network parameters. , , , ; Step 4.8: The training process from Step 4.2 to Step 4.7 is carried out in an end-to-end manner. By continuously interacting with the environment, experience is accumulated and the parameters of each network are updated, and finally a safe and efficient obstacle avoidance and tracking strategy that can be implemented in complex electromagnetic environments is obtained. The target UAV is tracked and tracked based on the optimal obstacle avoidance and tracking strategy.

Images