Obstacle avoidance tracking method based on deep reinforcement learning

Abstract

The invention discloses an obstacle avoidance tracking method based on deep reinforcement learning, and belongs to the field of path planning. The invention relates to a multi-unmanned aerial vehicle reconnaissance near-end strategy optimization method based on a distributed reinforcement learning framework, and the implementation method is as follows: through deep fusion of a pressure sensing mechanism and an improved SAC reinforcement learning method, precise avoidance of dynamic obstacles and efficient pursuit of abnormal radiation sources are realized. According to the pressure sensing module designed by the invention, the environmental pressure is quantized into a numerical index, and the conservative degree of a decision strategy is dynamically adjusted according to the numerical index. On this basis, decision optimization is carried out by using a double-Q network architecture of an improved SAC algorithm, and a conservative strategy and an aggressive strategy are adopted for parallel evaluation, so that the robustness of the strategies is improved while the exploration efficiency is ensured. The method operates in a continuous action space, and the smoothness and stability of the motion trail of the unmanned aerial vehicle are ensured. According to the invention, the unmanned aerial vehicle can intelligently track an abnormal radiation source in a high-voltage environment in complex urban electromagnetic spectrum combat.

Description

Technical Field

[0001] The present invention relates to an obstacle avoidance and tracking method based on reinforcement learning, and belongs to the field of path planning. Technical Background

[0002] With the rapid development of modern cities, the urban electromagnetic environment has become unprecedentedly complex. Electromagnetic radiation generated by various communications equipment, industrial equipment, and consumer electronics interweaves, creating a dense and complex electromagnetic spectrum environment. The increasing adoption of drone technology has exacerbated this situation. On the one hand, the widespread use of commercial and consumer drones has generated a large number of new electromagnetic signal sources. On the other hand, the demand for drones to detect and track anomalous radiation sources is increasing in law enforcement, security, and other fields. Rapidly tracking suspicious electromagnetic signal sources has become a critical task in major event security. However, since our monitoring equipment and ordinary, legitimate drones operate in the same electromagnetic space, signal interference is highly likely to occur. This "electromagnetic accidental damage" not only impacts mission effectiveness but can also render legitimate equipment inoperable. Therefore, in the complex urban electromagnetic environment, developing intelligent obstacle avoidance and tracking systems has become a core task in electromagnetic spectrum warfare.

[0003] Currently, traditional methods for UAV path planning in urban environments primarily include sampling-based, search-based, and optimization-based approaches. Sampling-based planning (e.g., RRT and PRM) is highly efficient in processing high-dimensional spaces and possesses probabilistic completeness, but suffers from less than smooth paths and difficulty guaranteeing optimality. Search-based planning (e.g., A* and D*) is theoretically complete and can guarantee optimal paths, but suffers from the curse of dimensionality and high storage overhead. Optimization-based planning (e.g., MPC) can generate smooth trajectories and easily incorporate constraints, but is prone to local optimality and exhibits high computational complexity. Reinforcement learning algorithms, through end-to-end learning and continuous exploration and optimization, can adapt to complex dynamic environments, integrate multi-source sensory information, and improve computational efficiency with experience. In particular, in complex urban electromagnetic environments, they can learn signal propagation characteristics and environmental interaction patterns, balance exploration and exploitation to avoid local optimality, dynamically adjust decision strategies to improve robustness, and reduce the online computational burden to enhance real-time performance, effectively overcoming the limitations of traditional methods.

[0004] However, traditional reinforcement learning suffers from shortcomings such as low sample efficiency, unstable training, and poor policy generalization. Furthermore, exploration efficiency is low in high-dimensional action spaces, and reward function design struggles to balance multi-objective requirements, making it challenging to ensure real-time performance and accuracy. In real-world drone tracking tasks, the environment state space is vast, making traditional reinforcement learning difficult to quickly adapt to unexpected situations. Therefore, designing resilient reinforcement learning algorithms to enhance drone path planning capabilities in high-pressure environments, effectively targeting anomalous radiation sources, and balancing exploration efficiency with decision stability remains a pressing challenge. Summary of the Invention

[0005] In order to solve the problem of difficulty in achieving stable pursuit of abnormal radiation sources in high-pressure environments, the purpose of the present invention is to provide an obstacle avoidance and tracking method based on deep reinforcement learning. Through the deep integration of the pressure perception mechanism and the improved SAC (Soft Actor-Critic) reinforcement learning method, the precise avoidance of dynamic obstacles and the efficient pursuit of abnormal radiation sources are achieved. The pressure perception module designed by the present invention quantifies the environmental pressure into a numerical indicator and dynamically adjusts the conservatism of the decision-making strategy accordingly. On this basis, the dual-Q network architecture of the improved SAC algorithm is used for decision optimization, and conservative and aggressive strategies are evaluated in parallel to ensure exploration efficiency while improving the robustness of the strategy. The present invention operates in a continuous action space to ensure the smoothness and stability of the UAV's motion trajectory. The present invention can effectively solve the challenge of intelligent pursuit of abnormal radiation sources by UAVs in high-pressure environments in complex urban electromagnetic spectrum operations, and realize abnormal radiation source tracking based on elastic reinforcement learning.

[0006] The purpose of the present invention is achieved through the following technical solutions.

[0007] The present invention discloses an obstacle avoidance and tracking method based on deep reinforcement learning, which is a multi-UAV reconnaissance proximal strategy optimization method based on a distributed reinforcement learning framework, comprising the following steps:

[0008] Step 1: Build a multi-layer environment perception system to obtain environmental status information. The multi-layer environment perception system includes a terrain layer, an electromagnetic layer, and a pressure layer.

[0009] Step 2: Build a layered elastic obstacle avoidance decision-making framework, consisting of a pressure perception module, a value assessment module, and a policy decision-making module. The pressure perception module performs risk assessment based on the integrated environmental pressure output by the pressure layer. The value assessment determines the state-action value under the current pressure state. The policy network generates decision instructions adapted to the current pressure based on the value assessment. These three modules form a closed-loop feedback mechanism.

[0010] Step 3: Design a dynamic reward mechanism. Design tracking rewards based on environmental status information, design risk aversion rewards based on pressure layer assessment, and dynamically adjust reward weights based on the pressure level.

[0011] Step 4: Based on step 2, a hierarchical elastic obstacle avoidance decision framework is constructed. The dynamic reward mechanism designed in step 3 is adopted to construct the optimal obstacle avoidance and tracking strategy under different pressure environments. The obstacle avoidance and tracking of the target UAV are achieved according to the optimal obstacle avoidance and tracking strategy.

[0012] Furthermore, the implementation method of step 1 is:

[0013] A unified two-dimensional coordinate system is established, and the coordinate system space is discretized into a grid to obtain a grid map.

[0014] The multi-layer environmental perception system is divided into three main layers: terrain layer, electromagnetic layer and pressure layer.

[0015] The terrain layer mainly contains the information of non-radiative obstacles in the city. Each obstacle o i Expressed as:

[0016] o i ={p i ,s i ,b i ,r i} (1)

[0017] where p i is the coordinate of the obstacle center, s i is the geometric size, b i is the electromagnetic wave blocking coefficient, r i is the risk factor. All obstacle information is mapped into the grid map to form the obstacle distribution matrix B:

[0018] B={b ij |i,j∈N} (2)

[0019] where b ij Represents the obstacle distribution information of the i-th row and j-th column in the grid map, and N represents the dimensional range of the grid map.

[0020] The electromagnetic layer contains k drones, and its electromagnetic radiation intensity model is:

[0021]

[0022] Among them, P k is the radiation power, (x k ,y k ) is the position of the UAV, σ k is the radiation range parameter.

[0023] The pressure layer introduces a pressure metric P to quantify the threat level of the current environment:

[0024] P=α1·D t +β1·E i +γ1·R o (4)

[0025] Among them D t is the distance pressure from the target, E i is the environmental interference pressure, R o is the obstacle risk pressure, and α1, β1, and γ1 are the corresponding weight coefficients.

[0026] The overall environmental state distribution considering the obstacle blocking effect and pressure influence is:

[0027]

[0028] The occlusion coefficient μ(x,y,z) is determined by the obstacle distribution matrix B:

[0029] μ(x,y)=f(B,x,y) (6)

[0030] The pressure regulation function ψ(P) is:

[0031]

[0032] Where P0 is the pressure threshold and λ is the pressure adjustment sensitivity coefficient.

[0033] Furthermore, the state obtained in step 1 is expressed as:

[0034] s t ={B,E,P} (8)

[0035] Furthermore, the pressure sensing module in step 2 is obtained based on the pressure index P of the pressure layer in step 1.

[0036] The pressure assessment function of the pressure perception module is expressed as:

[0037]

[0038] Where τ is the time variable used for pressure integral calculation. w1, w2, and w3 are weight coefficients corresponding to pressure value, pressure change rate, and pressure accumulation effect, respectively. p It is the pressure assessment value, which is used to comprehensively assess the environmental pressure. p Dynamically score the stress state.

[0039] Furthermore, the value evaluation module in step 2 designs a dynamic balanced dual Q network structure, including a conservative value network Q c and Radical Value Network Q a :

[0040] When the ambient pressure value V p >0.5:

[0041] Q c (s t ,a t θ c )=E[r t (1+α2V p )+γQ c (s t+1 ,a t+1θ c′ )] (10)

[0042] When the ambient pressure value V p When ≤0.5:

[0043] Q a (s t ,a t θ a )=E[r t (1-β2V p )+γQ a (s t+1 ,a t+1 θ a′ )] (11)

[0044] α2 and β2 are pressure influence coefficients, which are adjusted by adjusting the immediate reward r t , so that the conservative network gets a larger weight under high pressure and the radical network gets a larger weight under low pressure.

[0045] When V p When >0.5, the loss function is defined as:

[0046]

[0047] When V p When ≤0.5, the loss function is defined as:

[0048]

[0049] The target value y a 、y c Calculated as follows:

[0050] y c =r t (1+α2V p )+γ·Q c (s t+1 ,a t+1 θ c′ ) (14)

[0051] y a =r t (1-β2V p )+γ·Q a (s t+1 ,a t+1 θ a′ ) (15)

[0052] Where γ is the discount factor and α is the entropy regularization coefficient.

[0053] Furthermore, the action decision module in step 2 dynamically selects the strategy network output according to the environmental pressure value:

[0054] When the ambient pressure value V p When >0.5, a conservative strategy network is used:

[0055]

[0056] where π c represents a conservative policy network. c represents the conservative policy network parameters. τ is the temperature parameter that controls the degree of exploration of the policy. a′ represents all possible actions in the action space.

[0057] When the ambient pressure value V p When ≤0.5, the aggressive strategy network is adopted:

[0058]

[0059] where π a represents the aggressive strategy network, φ a Represents the aggressive strategy network parameters.

[0060] The loss function of the policy network is V p When >0.5, it is defined as:

[0061]

[0062] In V p When ≤0.5:

[0063]

[0064] where ρ represents the state distribution. represents the expectation under the state distribution ρ. α is the entropy regularization coefficient, which is used to balance exploration and exploitation.

[0065] Furthermore, the implementation method of step three is:

[0066] The basic tracking reward considering the target distance and closing speed is expressed as:

[0067]

[0068] where d t represents the current distance to the target, η is the speed reward coefficient, and Δt is the time interval.

[0069] The obstacle avoidance safety reward is a penalty for the obstacle distance in the form of an exponential function, which is called the obstacle avoidance safety reward and is expressed as:

[0070]

[0071] where k s represents the obstacle avoidance sensitivity coefficient, d obs Indicates the distance to the obstacle, d safe Indicates the safety distance threshold.

[0072] At the same time, the pressure regulation reward r is introduced press =-P·|a t | 2 and stability reward r stab =-|a t -a t-1 | 2 , respectively constraining the system's behavior amplitude and action continuity under high pressure environment.

[0073] The final comprehensive reward function achieves adaptive integration of various rewards through dynamic weight coefficients:

[0074] r t =κ1·r track +κ2·r safe +κ3·r press +κ4·r stab (twenty two)

[0075] Furthermore, the implementation method of step 4 is:

[0076] Step 4.1: Initialize the conservative value network parameters θ c , radical value network parameter θ a , conservative strategy network parameter φ c , aggressive strategy network parameter φ a The initial environmental state s0 and pressure value V p =0.

[0077] Step 4.2: At the beginning of each training round, reset the environment state s t and pressure value V p =0.

[0078] Step 4.3: Based on the current environment state s t and pressure value V p , sampling actions from the corresponding policy network.

[0079] Step 4.4: Execute action a t , the next environment state s t+1 . And get the reward r set in step 3 t .

[0080] Step 4.5: Update the value network and policy network parameters based on the status and reward from step 4.4.

[0081] Step 4.6: After reaching the goal state or exceeding the maximum number of steps, the current round ends.

[0082] Step 4.7: Reset the environment state and pressure value to make the environment state s0, V p = 0. Repeat training steps 4.3 to 4.6 until convergence. Output network parameters θ c ,θ a 、φ c 、φ a .

[0083] Step 4.8: The training process from step 4.2 to step 4.7 is performed in an end-to-end manner. By continuously interacting with the environment, accumulating experience and updating the parameters of each network, a safe and efficient obstacle avoidance and tracking strategy is finally obtained in a complex electromagnetic environment. The target UAV is tracked according to the optimal obstacle avoidance and tracking strategy.

[0084] Beneficial effects:

[0085] This invention discloses an obstacle avoidance and tracking method based on deep reinforcement learning. By constructing a multi-layered environmental perception system, it decomposes complex adversarial environments into three dimensions: obstacle layer, electromagnetic layer, and pressure layer. This method achieves comprehensive perception and quantitative expression of environmental threats. In particular, the introduction of a pressure metric, P, effectively integrates distance pressure, environmental interference pressure, and obstacle risk pressure, enabling the system to accurately assess the current situation and provide a reliable basis for subsequent decision-making.

[0086] 2. This invention discloses an obstacle avoidance and tracking method based on deep reinforcement learning. The designed hierarchical elastic obstacle avoidance decision-making framework is highly adaptable and robust. A dual-Q network structure enables dynamic switching between conservative and aggressive assessments, and a pressure adjustment factor is used to adaptively adjust strategy selection, enabling the drone to flexibly select an avoidance strategy based on environmental pressure. Furthermore, based on strategy selection using an improved SAC algorithm, an adaptive temperature parameter is used to achieve a dynamic balance between exploration and exploitation, improving decision reliability.

[0087] 3. This invention discloses an obstacle avoidance and tracking method based on deep reinforcement learning. Its dynamic reward mechanism comprehensively considers tracking performance, obstacle avoidance safety, pressure response, and control stability. The weight coefficients of each reward component can be dynamically adjusted based on the pressure state. This method ensures the safety of drones while improving tracking efficiency. It achieves rapid response to environmental pressure while maintaining control stability, significantly enhancing the overall performance of drones in complex adversarial environments.

[0088] 4. The obstacle avoidance and tracking method based on deep reinforcement learning disclosed in the present invention, on the basis of achieving the above three beneficial effects, adopts a UAV reinforcement learning method based on a layered elastic framework. Through the deep integration of the pressure perception mechanism and the improved SAC (Soft Actor-Critic) reinforcement learning method, it can effectively solve the challenge of UAVs intelligently pursuing abnormal radiation sources in high-pressure environments in complex urban electromagnetic spectrum operations, and realize abnormal radiation source tracking based on elastic reinforcement learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] Figure 1 Schematic diagram of terrain perception.

[0090] Figure 2 Schematic diagram of electromagnetic sensing.

[0091] Figure 3 This is a flow chart of the reinforcement learning obstacle avoidance and tracking algorithm disclosed in the present invention. DETAILED DESCRIPTION

[0092] In order to enable those skilled in the art to have a deeper understanding of the implementation ideas of the solutions of the present invention, the technical solutions in the embodiments of the present invention will be described in detail and clearly in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other implementation cases obtained by ordinary technicians in this field without making creative work should fall within the scope of protection of the present invention. The specific steps of the embodiments of the present invention will be described below in conjunction with specific scenarios.

[0093] Example:

[0094] The abnormal radiation source tracking scenario in this embodiment is as follows Figure 1 As shown, a multi-layered environmental perception system for multi-agent collaboration is constructed over a 10 km x 10 km area, deploying three Red jammers, two White drones, and one Blue drone. During each mission evolution, the system records key data. This invention considers collaborative multi-agent localization of anomalous radiation sources.

[0095] This embodiment discloses an obstacle avoidance and tracking method based on deep reinforcement learning. The specific implementation steps are as follows:

[0096] Step 1: Build a multi-layer environmental perception system, including terrain layer, electromagnetic layer and pressure layer, to obtain environmental status information.

[0097] First, a unified two-dimensional coordinate system is established, and the coordinate system space is discretized into a grid to obtain a grid map.

[0098] The multi-layer environmental perception system is divided into three main layers: terrain layer, electromagnetic layer and pressure layer.

[0099] like Figure 1 As shown, the terrain layer mainly contains the information of non-radiative obstacles in the city. i Expressed as:

[0100] o i ={p i ,s i ,b i ,r i} (twenty three)

[0101] where p i is the coordinate of the obstacle center, s i is the geometric size, b i is the electromagnetic wave blocking coefficient, r i is the risk factor. All obstacle information is mapped into the grid map to form the obstacle distribution matrix B:

[0102] B={b ij |i,j∈N} (24)

[0103] where b ij Represents the obstacle distribution information of the i-th row and j-th column in the grid map, and N represents the dimensional range of the grid map.

[0104] like Figure 2 As shown in the figure, the electromagnetic layer contains 6 drones, and the electromagnetic radiation intensity model of the kth drone is:

[0105]

[0106] Among them, P k is the radiation power, (x k ,y k ) is the position of the UAV, σ k is the radiation range parameter.

[0107] The pressure layer introduces a pressure metric P to quantify the threat level of the current environment:

[0108] P=α1·D t +β1·E i +γ1·R o (26)

[0109] Among them D t is the distance pressure from the target, E i is the environmental interference pressure, R o is the obstacle risk pressure, α1, β1, and γ1 are the corresponding weight coefficients, which are set to 0.4, 0.3, and 0.3 respectively.

[0110] The overall environmental state distribution considering the obstacle blocking effect and pressure influence is:

[0111]

[0112] The occlusion coefficient μ(x,y,z) is determined by the obstacle distribution matrix B:

[0113] μ(x,y)=f(B,x,y) (28)

[0114] The pressure regulation function ψ(P) is:

[0115]

[0116] Where P0 is the pressure threshold and λ is the pressure adjustment sensitivity coefficient, which are set to 0 and 1 respectively.

[0117] The overall status is expressed as:

[0118] s t ={B,E,P} (30)

[0119] Step 2: Build a layered elastic obstacle avoidance decision-making framework, consisting of a pressure perception module, a value assessment module, and a policy decision-making module. The pressure perception module performs risk assessment based on the integrated environmental pressure output by the pressure layer. The value assessment determines the state-action value under the current pressure state. The policy network generates decision instructions adapted to the current pressure based on the value assessment. These three modules form a closed-loop feedback mechanism.

[0120] The pressure assessment function of the pressure perception module is expressed as:

[0121]

[0122] Where τ is the time variable used for pressure integral calculation. w1, w2, and w3 are weight coefficients corresponding to pressure value, pressure change rate, and pressure accumulation effect, respectively, and are set to 0.4, 0.3, and 0.3. p It is the pressure assessment value, which is used to comprehensively assess the environmental pressure. p Dynamically score the stress state.

[0123] The value assessment module designs a dynamic balanced dual Q network structure, including the conservative value network Q c and Radical Value Network Q a :

[0124] When the ambient pressure value V p >0.5:

[0125] Q c (s t ,a t θ c)=E[r t (1+α2V p )+γQ c (s t+1 ,a t+1 θ c′ )] (32)

[0126] When the ambient pressure value V p When ≤0.5:

[0127] Q a (s t ,a t θ a )=E[r t (1-β2V p )+γQ a (s t+1 ,a t+1 θ a′ )] (33)

[0128] α2 and β2 are pressure influence coefficients, both set to 1. By adjusting the immediate reward r t , so that the conservative network gets a larger weight under high pressure and the radical network gets a larger weight under low pressure.

[0129] When V p When >0.5, the loss function is defined as:

[0130]

[0131] When V p When ≤0.5, the loss function is defined as:

[0132]

[0133] The target value y a 、y c Calculated as follows:

[0134] y c =r t (1+α2V p )+γ·Q c (s t+1 ,a t+1 θ c′ ) (36)

[0135] y a =r t (1-β2V p )+γ·Q a (s t+1 ,a t+1 θ a′ ) (37)

[0136] Where γ is the discount factor, which is set to 0.8, and α is the entropy regularization coefficient, which is set to 0.1.

[0137] The policy decision module dynamically selects the policy network output according to the environmental pressure value:

[0138] When the ambient pressure value V p When >0.5, a conservative strategy network is used:

[0139]

[0140] where π c represents a conservative policy network. c represents the conservative policy network parameters. τ is the temperature parameter that controls the degree of exploration of the policy and is set to 0.5. a′ represents all possible actions in the action space.

[0141] When the ambient pressure value V p When ≤0.5, the aggressive strategy network is adopted:

[0142]

[0143] where π a represents the aggressive strategy network, φ a Represents the aggressive strategy network parameters.

[0144] The loss function of the policy network is V p When >0.5, it is defined as:

[0145]

[0146] In V p When ≤0.5:

[0147]

[0148] where ρ represents the state distribution. represents the expectation under the state distribution ρ. α is the entropy regularization coefficient, which is set to 0.8 to balance exploration and exploitation.

[0149] Step 3: Design a dynamic reward mechanism. Design tracking rewards based on environmental status information, design risk aversion rewards based on pressure layer assessment, and dynamically adjust reward weights based on the pressure level.

[0150] The basic tracking reward considering the target distance and closing speed is expressed as:

[0151]

[0152] where d trepresents the current distance to the target, η is the speed reward coefficient, which is set to 1, and Δt is the time interval.

[0153] The obstacle avoidance safety reward is a penalty for the obstacle distance in the form of an exponential function, which is called the obstacle avoidance safety reward and is expressed as:

[0154]

[0155] where k s Indicates the obstacle avoidance sensitivity coefficient, set to 1, d obs Indicates the distance to the obstacle, d safe Indicates the safety distance threshold, set to 20 meters.

[0156] At the same time, the pressure regulation reward r is introduced press =-P·|a t | 2 and stability reward r stab =-|a t -a t-1 | 2 , respectively constraining the system's behavior amplitude and action continuity under high pressure environment.

[0157] The final comprehensive reward function achieves adaptive integration of various rewards through dynamic weight coefficients:

[0158] r t =κ1·r track +κ2·r safe +κ3·r press +κ4·r stab (44)

[0159] Among them, κ1, κ2, κ3, and κ4 are all set to 0.25.

[0160] Step 4: Execute the algorithm training and testing process to achieve the optimal obstacle avoidance and tracking strategy under different pressure environments.

[0161] Step 4.1: Initialize the conservative value network parameters θ c , radical value network parameter θ a , conservative strategy network parameter φ c , aggressive strategy network parameter φ a The initial environmental state s0 and pressure value V p =0.

[0162] Step 4.2: At the beginning of each training round, reset the environment state s t and pressure value V p =0.

[0163] Step 4.3: Based on the current environment state s t and pressure value Vp , sampling actions from the corresponding policy network.

[0164] Step 4.4: Execute action a t , the next environment state s t+1 . And get the reward r set in step 3 t .

[0165] Step 4.5: Update the value network and policy network parameters based on the status and reward from step 4.4.

[0166] Step 4.6: After reaching the goal state or exceeding the maximum number of steps, the current round ends.

[0167] Step 4.7: Reset the environment state and pressure value to make the environment state s0, V p = 0. Repeat training steps 4.3 to 4.6 until convergence. Output network parameters θ c ,θ a 、φ c 、φ a .

[0168] Step 4.8: The training process from step 4.2 to step 4.7 is performed in an end-to-end manner. By continuously interacting with the environment, accumulating experience and updating the parameters of each network, a safe and efficient obstacle avoidance and tracking strategy is finally obtained in a complex electromagnetic environment. The target UAV is tracked according to the optimal obstacle avoidance and tracking strategy.

[0169] Step 5: During the testing phase, perform the test according to the following process:

[0170] Step 5.1: Load the network parameters θ trained in step 4 c ,θ a 、φ c 、φ a .

[0171] Step 5.2: Get the current environment status s t and pressure value V p .

[0172] Step 5.3: According to the pressure value V p Select the corresponding policy network and determine the execution action.

[0173] Step 5.4: Execute the action and obtain the new state s t+1 and pressure value V p .

[0174] Step 5.5: Repeat test steps 5.2 to 5.4. If the target is caught, the test is successful. Otherwise, the test fails and needs to be retrained in the new environment.

[0175] The above specific description further illustrates the purpose, technical solutions and beneficial effects of the invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An obstacle avoidance and tracking method based on deep reinforcement learning, characterized by: The following steps are included: Step 1: Build a multi-layer environmental perception system to obtain environmental status information; the multi-layer environmental perception system includes a terrain layer, an electromagnetic layer, and a pressure layer; Step 2: Build a layered elastic obstacle avoidance decision-making framework, which includes a pressure perception module, a value assessment module, and a policy decision-making module. The pressure perception module performs risk assessment based on the comprehensive environmental pressure output by the pressure layer, and the value assessment determines the state-action value under the current pressure state. The policy network generates decision instructions adapted to the current pressure based on the value assessment. These three modules form a closed-loop feedback mechanism. Step 3: Design a dynamic reward mechanism. Design tracking rewards based on environmental status information, design risk aversion rewards based on pressure level assessment, and dynamically adjust reward weights based on pressure levels. Step 4: Based on step 2, a hierarchical elastic obstacle avoidance decision framework is constructed. The dynamic reward mechanism designed in step 3 is adopted to construct the optimal obstacle avoidance and tracking strategy under different pressure environments. The obstacle avoidance and tracking of the target UAV are achieved according to the optimal obstacle avoidance and tracking strategy.

2. The obstacle avoidance and tracking method based on deep reinforcement learning according to claim 1, characterized in that: The implementation method of step one is: Establish a unified two-dimensional coordinate system and discretize the coordinate system space into a grid to obtain a grid map; The multi-layer environmental perception system is divided into three main layers: terrain layer, electromagnetic layer and pressure layer; The terrain layer mainly contains the information of non-radiative obstacles in the city; each obstacle o i Expressed as: o i ={p i ,s i ,b i ,r i } (1) where p i is the coordinate of the obstacle center, s i is the geometric size, b i is the electromagnetic wave blocking coefficient, r i is the risk coefficient; all obstacle information is mapped into the grid map to form the obstacle distribution matrix B: B={b ij |i,j∈N} (2) where b ij Represents the obstacle distribution information of the i-th row and j-th column in the grid map, and N represents the dimensional range of the grid map; The electromagnetic layer contains k drones, and its electromagnetic radiation intensity model is: Among them, P k is the radiation power, (x k ,y k ) is the position of the UAV, σ k is the radiation range parameter; The pressure layer introduces a pressure metric P to quantify the threat level of the current environment: P=α1·D t +β1·E i +γ1·R o (4) Among them D t is the distance pressure from the target, E i is the environmental interference pressure, R o is the obstacle risk pressure, α1, β1, γ1 are the corresponding weight coefficients; The overall environmental state distribution considering the obstacle blocking effect and pressure influence is: The occlusion coefficient μ(x,y,z) is determined by the obstacle distribution matrix B: μ(x,y)=f(B,x,y) (6) The pressure regulation function ψ(P) is: Where P0 is the pressure threshold and λ is the pressure adjustment sensitivity coefficient.

3. The obstacle avoidance and tracking method based on deep reinforcement learning according to claim 1, characterized in that: The state obtained in step 1 is expressed as: s t ={B,E,P} (8)。 4. The obstacle avoidance and tracking method based on deep reinforcement learning according to claim 1, characterized in that: The pressure sensing module in step 2 is obtained based on the pressure index P of the pressure layer in step 1; The pressure assessment function of the pressure perception module is expressed as: Among them, τ is the time variable used for pressure integral calculation; w1, w2, w3 are weight coefficients corresponding to pressure value, pressure change rate and pressure accumulation effect respectively; V p is the pressure assessment value, which is used to comprehensively assess the environmental pressure; according to the assessment value V p Dynamically score the stress state.

5. The obstacle avoidance and tracking method based on deep reinforcement learning according to claim 1, characterized in that: The value evaluation module in step 2 designs a dynamic balanced dual Q network structure, including the conservative value network Q c and Radical Value Network Q a : When the ambient pressure value V p >0.5: Q c (s t ,a t ;θ c )=E[r t (1+α2V p )+γQ c (s t+1 ,a t+1 ;θ c′ )] (10) When the ambient pressure value V p When ≤0.5: Q a (s t ,a t ;θ a )=E[r t (1-β2V p )+γQ a (s t+1 ,a t+1 ;θ a′ )] (11) α2 and β2 are pressure influence coefficients, which are adjusted by adjusting the immediate reward r t , so that the conservative network gets more weight under high pressure and the radical network gets more weight under low pressure; When V p When >0.5, the loss function is defined as: When V p When ≤0.5, the loss function is defined as: The target value y is calculated as follows: y c =r t (1+α2V p )+γ·Q c (s t+1 ,a t+1 ;θ c′ ) (14) y a =r t (1-β2V p )+γ·Q a (s t+1 ,a t+1 ;θ a′ ) (15) Where γ is the discount factor and α is the entropy regularization coefficient.

6. The obstacle avoidance and tracking method based on deep reinforcement learning according to claim 1, characterized in that: The action decision module in step 2 dynamically selects the strategy network output according to the environmental pressure value: When the ambient pressure value V p When >0.5, a conservative strategy network is used: where π c represents a conservative policy network; φ c represents the conservative strategy network parameters; τ is the temperature parameter, which controls the exploration degree of the strategy; a′ represents all possible actions in the action space; When the ambient pressure value V p When ≤0.5, the aggressive strategy network is adopted: where π a represents the aggressive strategy network, φ a represents the parameters of the aggressive strategy network; The loss function of the policy network is V p When >0.5, it is defined as: In V p When ≤0.5: Where ρ represents the state distribution; represents the expectation under the state distribution ρ; α is the entropy regularization coefficient, which is used to balance exploration and utilization.

7. The obstacle avoidance and tracking method based on deep reinforcement learning according to claim 1, characterized in that: The implementation method of step three is: The basic tracking reward considering the target distance and closing speed is expressed as: where d t represents the current distance to the target, η is the speed reward coefficient, and Δt is the time interval; The obstacle avoidance safety reward is a penalty for the obstacle distance in the form of an exponential function, which is called the obstacle avoidance safety reward and is expressed as: where k s represents the obstacle avoidance sensitivity coefficient, d obs Indicates the distance to the obstacle, d safe Indicates the safety distance threshold; At the same time, the pressure regulation reward r is introduced press =-P·|a t | 2 and stability reward r stab =-|a t -a t-1 | 2 , respectively constraining the system's behavior amplitude and action continuity under high-pressure environment; The final comprehensive reward function achieves adaptive integration of various rewards through dynamic weight coefficients: r t =κ1·r track +κ2·r safe +κ3·r press +κ4·r stab (22).

8. The obstacle avoidance and tracking method based on deep reinforcement learning according to claim 1, characterized in that: The implementation method of step 4 is: Step 4.1: Initialize the conservative value network parameters θ c , radical value network parameter θ a , conservative strategy network parameter φ c , aggressive strategy network parameter φ a ; Initial environmental state s0 and pressure value V p =0; Step 4.2: At the beginning of each training round, reset the environment state s t and pressure value V p =0; Step 4.3: Based on the current environment state s t and pressure value V p , sample actions from the corresponding policy network; Step 4.4: Execute action a t , the next environment state s t+1 ; and obtain the reward r set in step 3 t ; Step 4.5: Update the value network and policy network parameters based on the status and reward of step 4.4; Step 4.6: After reaching the target state or exceeding the maximum number of steps, the current round ends; Step 4.7: Reset the environment state and pressure value to make the environment state s0, V p = 0; repeat training steps (3) to (6) until convergence; output network parameters θ c ,θ a 、φ c 、φ a ; Step 4.8: The training process from step 4.2 to step 4.7 is performed in an end-to-end manner. By continuously interacting with the environment, accumulating experience and updating the parameters of each network, a safe and efficient obstacle avoidance and tracking strategy is finally obtained in a complex electromagnetic environment. The target UAV is tracked according to the optimal obstacle avoidance and tracking strategy.

Images