A method for unmanned aerial vehicle path planning fuzz testing based on reinforcement learning

Abstract

The application discloses a kind of unmanned vehicle path planning fuzzy testing methods based on reinforcement learning, the method includes: constructing three-dimensional simulation test environment that can be disturbed, real-time reading the environment parameter of three-dimensional simulation test environment;Design test agent, test agent is based on environment parameter and executes state input, action output and basic reward function;Introduce path planning failure sensitivity weighting mechanism, and dynamically weight basic reward function;Strategy training and interactive test are carried out to test agent, update environment parameter;Store failure case in interactive test, introduce deep reinforcement learning transfer mechanism and carry out cross-scene transfer.The application can adaptively construct challenging test scene, accurately locate algorithm weakness, and provide interpretable feedback for algorithm iteration, thereby comprehensively improving the safety and reliability of unmanned system in real world.

Description

Technical Field

[0001] This invention belongs to the technical field of the intersection of drone safety detection and artificial intelligence, and relates to a fuzzy testing method for drone path planning based on reinforcement learning. Background Technology

[0002] With the widespread application of drones in logistics delivery, emergency rescue, agricultural plant protection, power line inspection, and security monitoring, their autonomous navigation and path planning capabilities have become one of the core technologies for ensuring the safe and efficient execution of missions. Path planning algorithms are responsible for generating a feasible trajectory between a given starting point and target point, taking into account environmental information such as obstacle distribution, no-fly zones, and dynamic threats, while satisfying multi-dimensional constraints such as safety, timeliness, and energy consumption. However, real-world application scenarios are often highly complex, uncertain, and dynamic. For example, GPS signal obstruction in urban canyons, the appearance of sudden obstacles, communication delays or interference, and sensor noise can all significantly affect the actual performance of path planning algorithms and even lead to flight accidents.

[0003] Currently, mainstream path planning methods include graph search-based algorithms such as A* and Dijkstra, sampling-based RRT and its variants, and end-to-end planning strategies based on deep learning and reinforcement learning that have emerged in recent years. While these methods demonstrate good performance under simulated or ideal conditions, their robustness and generalization ability remain significantly insufficient when facing unseen edge scenarios or adversarial perturbations. Traditional testing methods, such as rule-based scene replay, Monte Carlo random sampling, or manually designed boundary cases, often have limited coverage, are inefficient, and struggle to proactively uncover the algorithm's vulnerabilities.

[0004] Fuzz testing, a widely used automated testing technique in software security, triggers potential crashes, logical errors, or security vulnerabilities by inputting large amounts of abnormal, unexpected, or randomly constructed data into the system. Introducing fuzz testing into the field of UAV path planning hinges on how to intelligently generate meaningful perturbation environments or task conditions, rather than simple random perturbations. In recent years, reinforcement learning, with its advantages in sequential decision-making, environmental interaction, and policy optimization, has provided new ideas for building intelligent fuzz testing frameworks: a test agent can be trained to actively explore test scenarios most likely to lead to planning failure within a state space that includes obstacle layout, wind speed changes, and takeoff and landing point offsets, with the goal of maximizing the failure probability of the path planning algorithm under test.

[0005] However, current research in this area is still in its early stages, lacking systematic test framework design, effective reward mechanism definition, and the ability to model perturbations in high-dimensional continuous environments. Furthermore, ensuring that generated test cases are both realistically feasible and efficiently cover diverse failure modes remains a pressing technical challenge. Summary of the Invention

[0006] This invention aims to address the technical problems in existing technologies, such as the limited coverage of traditional testing methods, the inapplicability of general fuzzy testing techniques to continuous and discrete hybrid decision systems like path planning, and the lack of proactive exploration capabilities for dynamically optimizing testing strategies based on algorithm responses. This invention provides a reinforcement learning-based fuzzy testing method for UAV path planning, and the technical solution adopted is as follows: A fuzzy testing method for UAV path planning based on reinforcement learning, comprising the following steps: S1. Construct a perturbable three-dimensional simulation test environment and read the environmental parameters of the three-dimensional simulation test environment in real time; S2. Design a test agent, which executes state input, action output and basic reward function based on the environmental parameters; S3. Introduce a path planning failure sensitivity weighting mechanism to dynamically weight the basic reward function; S4. Perform policy training and interactive testing on the test agent, and update the environmental parameters; S5. Store the failed test cases in the interactive test and introduce a deep reinforcement learning transfer mechanism for cross-scenario transfer.

[0007] In one embodiment of the present invention, in step S1, the environmental parameters include: airspace boundary, static obstacles, dynamic interference sources, sensing noise model, and task parameters; The perception noise model is constructed by dividing the UAV's perception state into position, velocity, and obstacle distance, and using a hybrid model of Gaussian white noise and random walk drift. Let the actual state of the drone be represented as: (1) In formula (1), This represents the true complete state vector of the tested UAV. This represents the actual three-dimensional position coordinates of the tested drone. This represents the actual distance observation vector between the tested UAV and surrounding obstacles. This represents the actual three-dimensional velocity vector of the tested UAV. The observation state after injecting noise is represented as follows: (2) In formula (2), Indicates the observation status. Indicates location observation value, Represents the observed velocity value. Indicates the observed distance to the obstacle; For location observations, inject white noise that follows a zero-mean Gaussian distribution and a drift term that accumulates over time: (3) In formula (3), Indicates location observation value, This represents the actual three-dimensional position coordinates of the tested drone. Let represent a random variable that follows a normal distribution. The standard deviation of Gaussian noise at location is represented. The set range is , Denotes the random walk drift term, satisfying , The standard deviation of Gaussian white noise represents the random increment of a single-step drift. This indicates the time difference between the tested drone and the start of the test; For velocity observations, only Gaussian white noise is injected: (4) In formula (4), Represents the observed velocity value. This represents the true three-dimensional velocity vector of the tested UAV. This represents the standard deviation of the velocity Gaussian noise. The set range is ; For obstacle distance observations, only Gaussian white noise is injected: (5) In formula (5), This represents the observed distance to the obstacle. This represents the actual distance observation vector between the tested UAV and surrounding obstacles. This represents the standard deviation of Gaussian noise at the distance to the obstacle. The set range is ; The disturbance intensity is controlled by the perceived noise model.

[0008] In one embodiment of the present invention, step S2 includes: S21: The test agent receives the environmental configuration information for the current test round, including but not limited to the spatial coordinates of the starting point and the target point, the spatial position, quantity and geometric features of obstacles, the state of dynamic interference sources, the execution result of the algorithm under test in the previous test round and historical failure statistics. All state variables are normalized or standardized and then used as the input of the test agent. S22: The test agent outputs a set of disturbance instructions for modifying the test environment. The disturbance dimensions include spatial offset between the starting point and the target point, addition or deletion or position adjustment of static obstacles, activation of dynamic interference sources, setting of speed or trajectory parameters and adjustment of sensor noise intensity. The value range of each action component is limited to the range that ensures the physical rationality of the test scenario and the feasibility of the task. S23: Define the basic reward function based on the execution results of the path planning algorithm under test in a disturbed environment.

[0009] In one embodiment of the present invention, the basic reward function consists of three parts: a termination reward, a time penalty term, and a heuristic guiding term. The calculation of the basic reward function is expressed as follows: (6) In formula (6), Represents the basic reward function. Indicates the reward at the end of the round. This indicates the single-step time penalty. This indicates a heuristic-guided reward system; The termination reward is triggered only when the drone collides, flies out of the boundary, or safely reaches the destination. If environmental disturbances cause the tested drone to collide or fly out of the safe boundary, the planning failure is triggered, and the agent is given a high positive reward. If the drone safely reaches the destination, a negative penalty will be imposed. ; The single-step time penalty term is a negative constant, used to encourage the agent to find the scenario that causes the drone to fail as quickly as possible. The test agent is given a small penalty for each exploration step it performs. ; The heuristic-guided reward is used to provide intensive reward feedback based on the danger level of the drone, and is based on the minimum distance between the drone's flight trajectory and the obstacle. , This indicates the minimum distance between the drone's flight path and the obstacle. This represents the adjustment coefficient. , Represents a positive minimum constant. The closer the drone gets to the obstacle, the greater the base reward the test agent receives.

[0010] In one embodiment of the present invention, step S3 includes: S31. Define the potential hazard level of the current environment; S32. Construct a monotonically increasing dynamic weight function based on the potential hazard level; S33. Dynamically weight the basic reward function and output the weighted basic reward to guide the test agent to prioritize exploring high-risk scenarios that are more likely to expose algorithm defects.

[0011] In one embodiment of the present invention, step S31 includes: The calculation of the potential hazard level is expressed as follows: (7) In formula (7), Indicates the time of testing The potential danger level; This represents the non-negative normalized weighting coefficient of the obstacle density term. This represents the non-negative normalized weight coefficient of the dynamic disturbance term. This represents the non-negative normalized weight coefficient of the task distance term; This indicates the number of static obstacles in the current environment; This represents a reference upper limit for the number of obstacles, used for normalization to ensure that the value of this item is within a reasonable range. Indicates the speed of motion of the dynamic interference source; The upper limit of the dynamic disturbance velocity is used for normalization and reflects the maximum disturbance intensity in a typical scenario. This represents the starting position vector of the drone mission; This represents the target point position vector of a UAV mission; It represents a very small positive constant used to avoid zero denominators and ensure the stability of mathematical expressions.

[0012] In one embodiment of the present invention, step S32 includes: The dynamic weighting function is calculated as follows: (8) In formula (8), Represents the dynamic weighting function. This represents the danger threshold, used to control the switching sensitivity of the weighting function. Indicates the time of testing The potential danger level.

[0013] In one embodiment of the present invention, step S33 includes: During testing, the hazard threshold was designed as a dynamically adaptive baseline threshold, with the threshold value updated in real time to reflect previous data. The moving average of potential hazards in historical testing. If the current environmental state is calculated hour, The tolerance constant representing the risk level is used to fine-tune the sensitivity of triggering high-risk scenario judgments. The system explicitly determines that the current test case has entered the high-risk scenario range. (Hyperbolic tangent term) The value increases rapidly, causing the dynamic weighting function to generate a gain factor greater than 1, approaching 2 at its highest, significantly amplifying the base reward; conversely, if the current area is low-risk, If so, the weight gain approaches 1 or even decays; The basic reward function is dynamically weighted to output the final reward.

[0014] In one embodiment of the present invention, step S4 includes: S41. Initialize the policy network and value evaluation network. The network structure adopts a multi-layer fully connected neural network. The hidden layer dimension and activation function are selected according to the task complexity. S42. Set training hyperparameters to prevent the test strategy from failing due to excessively large single parameter update magnitude, and ensure that the test agent can robustly converge to the high-risk perturbation strategy. S43. In each round of testing, the test agent generates a disturbance action based on the current policy, applies the disturbance in the simulation environment, calls the path planning algorithm under test, executes the generated flight trajectory, and determines the task result and calculates the basic reward. If the task result is a failure, a high positive reward is given. If the drone safely reaches the destination, a negative penalty is given. At the same time, the interaction sample is stored in the experience replay buffer. S44. After the test accumulates a sufficient number of interaction samples, the environmental parameters are updated according to the strategy until the preset termination condition is reached.

[0015] In one embodiment of the present invention, step S5 includes: S51. For each test round that triggers failure in the interactive test, the system automatically saves the complete environment configuration, the output trajectory of the algorithm under test, the failure type label and related metadata, stores them in a structured data format, and supports visual playback. S52. In the initialization phase of the target new scenario, the parameters of the policy network and value network that have been trained and converged in the source scenario are directly loaded. The gradient update of the feature extraction layer in the front end of the network is frozen based on the transfer learning theory to retain the prior knowledge of the test agent on the general dynamics and basic spatial avoidance logic of the UAV, and the network fine-tuning authority is strictly limited to the end action decision layer. S53. Limit the sampling size of the experience replay buffer during the exploration phase of the target scene, and set the sample size for a single iteration of exploration of the new scene. Standard interaction steps only for the source scene of , serving as a benchmark for interactive exploration of the target domain; S54. Employ a small-step parameter update strategy to fine-tune the learning rate for the target scenario. Set as the base learning rate of the source domain 0.1 times the value, and perform a maximum of M rounds of restricted gradient descent iterations.

[0016] The beneficial effects of this invention are: This invention presents a fuzzy testing method for UAV path planning based on reinforcement learning. By constructing an intelligent adversarial testing environment, it utilizes reinforcement learning agents to actively explore the safety boundaries and failure modes of path planning algorithms in complex, dynamic, or adversarial scenarios. Furthermore, it leverages cross-scenario transfer mechanisms to enhance the generalization ability and efficiency of testing. This systematically evaluates the robustness and security of UAV path planning systems, providing technical support for the highly reliable deployment of unmanned systems in military, emergency, and civilian scenarios. Attached Figure Description

[0017] Figure 1 This is a flowchart of a fuzzy testing method for UAV path planning based on reinforcement learning provided in an embodiment of the present invention; Figure 2 This is a framework diagram of the perceived noise model provided in an embodiment of the present invention; Figure 3 This is a conceptual diagram of dynamically weighting basic rewards using a failure sensitivity weighting mechanism provided in this embodiment of the invention; Figure 4 This is a flowchart of the reward calculation process for introducing path planning failure sensitivity, provided in an embodiment of the present invention. Detailed Implementation

[0018] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0019] This invention provides a fuzzy testing method for UAV path planning based on reinforcement learning, as shown in the appendix. Figure 1 The steps include: S1. Construct a perturbable 3D simulation test environment and read the environmental parameters of the 3D simulation test environment in real time; S2. Design a test agent that executes state input, action output, and basic reward function based on environmental parameters. S3. Introduce a path planning failure sensitivity weighting mechanism to dynamically weight the basic reward function; S4. Perform policy training and interactive testing on the test agent, and update the environment parameters; S5 stores failed test cases in interactive tests and introduces a deep reinforcement learning transfer mechanism for cross-scenario transfer.

[0020] This invention presents a fuzzy testing method for UAV path planning based on reinforcement learning. It can adaptively construct challenging test scenarios, accurately locate algorithm weaknesses, and provide interpretable feedback for algorithm iteration, thereby comprehensively improving the safety and reliability of unmanned systems in the real world.

[0021] Before each test, a high-fidelity drone simulation environment that supports dynamic parameter injection is built.

[0022] In step S1, the environmental parameters include: airspace boundary, static obstacles, dynamic interference sources, perception noise model, and task parameters.

[0023] Airspace boundary: defined as a three-dimensional bounded area, the size of which can be set within a reasonable range according to typical application scenarios such as urban, outdoor and indoor environments.

[0024] Static obstacles: The number, shape, size and spatial distribution can be configured. The types of obstacles include, but are not limited to, cubes, cylinders or polygonal entities, and their geometric parameters and density are within a reasonable range covered by typical flight missions.

[0025] Dynamic interference sources include moving obstacles or simulated environmental disturbance nodes, whose movement speed and trajectory patterns can be dynamically set.

[0026] Perception Noise Model: This model supports injecting controllable random noise into observation data from commonly used UAV sensors. The noise intensity can be adjusted within a range that does not affect the basic operability of the system. Specifically, the perception noise model divides the UAV's perception state into position, velocity, and obstacle distance, and constructs a hybrid model using Gaussian white noise and random walk drift. It takes the actual sensor state as input and outputs the perception state. See the framework diagram of the perception noise model for reference. Figure 2 .

[0027] Let the actual state of the drone be represented as: (1) In formula (1), This represents the true complete state vector of the tested UAV. This represents the actual three-dimensional position coordinates of the tested drone. This represents the actual distance observation vector between the tested UAV and surrounding obstacles. This represents the actual three-dimensional velocity vector of the tested UAV. The observation state after injecting noise is represented as follows: (2) In formula (2), Indicates the observation status. Indicates location observation value, Represents the observed velocity value. Indicates the observed distance to the obstacle; For location observations, inject white noise that follows a zero-mean Gaussian distribution and a drift term that accumulates over time: (3) In formula (3), Indicates location observation value, This represents the actual three-dimensional position coordinates of the tested drone. Let represent a random variable that follows a normal distribution. The standard deviation of Gaussian noise at location is represented. The set range is , Denotes the random walk drift term, satisfying , The standard deviation of Gaussian white noise represents the random increment of a single-step drift. This indicates the time difference between the tested drone and the start of the test; For velocity observations, only Gaussian white noise is injected: (4) In formula (4), Represents the observed velocity value. This represents the true three-dimensional velocity vector of the tested UAV. This represents the standard deviation of the velocity Gaussian noise. The set range is ; For obstacle distance observations, only Gaussian white noise is injected: (5) In formula (5), This represents the observed distance to the obstacle. This represents the actual distance observation vector between the tested UAV and surrounding obstacles. This represents the standard deviation of Gaussian noise at the distance to the obstacle. The set range is ; The disturbance intensity can be precisely controlled within a range that does not affect the basic operability of the system by using a perceived noise model.

[0028] Task parameters: The starting point and target point are set randomly or as needed in the airspace, and the distance between them meets the task feasibility requirements.

[0029] All environmental parameters are read and modified in real time through the program interface provided by the simulation platform, providing an operable perturbation dimension for the test agent.

[0030] Compared to traditional testing methods that rely on fixed test cases, random perturbations, or manual design, this invention enables the testing agent to actively explore high-dimensional environmental spaces, systematically discovering potential failure modes of path planning algorithms in edge scenarios and complex interference, achieving a more comprehensive characterization of security boundaries, and significantly improving the depth and breadth of test coverage.

[0031] The fuzz testing process is modeled as a Markov decision process, and the operations performed by the testing agent at each decision step include: S21: The test agent receives the environmental configuration information for the current test round, including but not limited to the spatial coordinates of the starting point and the target point, the spatial position, quantity and geometric characteristics of obstacles, the state of dynamic interference sources, the execution results of the algorithm under test in the previous test round and historical failure statistics. All state variables are normalized or standardized and then used as input to the test agent.

[0032] S22: The test agent outputs a set of perturbation instructions to modify the test environment. The perturbation dimensions include spatial offset of the starting point and the target point, addition, deletion or position adjustment of static obstacles, activation of dynamic interference sources, setting of speed or trajectory parameters and adjustment of sensor noise intensity. The value range of each action component is limited to the range that ensures the physical rationality of the test scenario and the feasibility of the task.

[0033] To address the characteristics of path planning systems, which combine continuous states and discrete decision-making, this invention designs a specialized state encoding mechanism and perturbation generation strategy. This enables fuzz testing to not only inject semantically reasonable environmental perturbations but also ensure that the test scenario conforms to physical laws and task logic, thus avoiding the generation of invalid or infeasible false test cases.

[0034] S23: Based on the execution results of the path planning algorithm under test in a perturbed environment, define the basic reward function. The basic reward function consists of three parts: a termination reward, a time penalty term, and a heuristic guidance term. The calculation of the basic reward function is expressed as follows: (6) In formula (6), Represents the basic reward function. Indicates the reward at the end of the round. This indicates the single-step time penalty. This indicates a heuristic-guided reward system.

[0035] Termination of reward is triggered only when the drone collides, flies out of the boundary, or safely reaches the destination. If current environmental disturbances cause the tested drone to collide or fly out of the safe boundary, triggering planning failure, the agent will be given a high positive reward. If the drone safely reaches the destination, a negative penalty will be imposed. ; The single-step time penalty term is a negative constant, designed to encourage the agent to quickly find the scenario that causes the drone to fail. A small penalty is applied to each exploration step performed by the agent during testing. ; Heuristic-guided rewards are used to provide intensive reward feedback based on the perceived danger level of the drone, derived from the minimum distance between the drone's flight path and obstacles. , This indicates the minimum distance between the drone's flight path and the obstacle. This represents the adjustment coefficient. , Represents a positive minimum constant. The closer the drone gets to the obstacle, the greater the base reward the testing agent receives.

[0036] To improve testing efficiency and avoid redundant exploration in low-risk areas, this invention introduces a failure sensitivity weighting mechanism to dynamically weight the basic reward, referring to... Figure 3 ,include: S31. Define the potential hazard level of the current environment; S32. Construct a monotonically increasing dynamic weight function based on the potential hazard level; S33. Dynamically weight the basic reward function and output the weighted basic reward to guide the test agent to prioritize exploring high-risk scenarios that are more likely to expose algorithm defects.

[0037] Define the potential hazard level of the current environment This indicator comprehensively considers factors reflecting environmental challenge, such as obstacle density, distance between start and end points, and intensity of dynamic disturbances, and is calculated through a weighted linear combination or nonlinear mapping. The potential hazard level is expressed as: (7) In formula (7), Indicates the time of testing The potential danger level; This represents the non-negative normalized weighting coefficient of the obstacle density term. This represents the non-negative normalized weight coefficient of the dynamic disturbance term. This represents the non-negative normalized weight coefficient of the task distance term; This indicates the number of static obstacles in the current environment; This represents a reference upper limit for the number of obstacles, used for normalization to ensure that the value of this item is within a reasonable range. Indicates the speed of motion of the dynamic interference source; The upper limit of the dynamic disturbance velocity is used for normalization and reflects the maximum disturbance intensity in a typical scenario. This represents the starting position vector of the drone mission; This represents the target point position vector of a UAV mission; It represents a very small positive constant used to avoid zero denominators and ensure the stability of mathematical expressions.

[0038] A monotonically increasing dynamic weight function is constructed based on the potential risk level, so that test samples corresponding to high-risk environments receive higher reward gains. The specific calculation of the dynamic weight function is expressed as follows: (8) In formula (8), Represents the dynamic weighting function. This represents the danger threshold, used to control the switching sensitivity of the weighting function. Indicates the time of testing The potential danger level.

[0039] The final reward is a weighted result, which guides the testing agent to prioritize exploring high-risk scenarios that are more likely to expose algorithmic flaws. The specific judgment criterion is: fully utilizing the risk threshold. .

[0040] During testing, the hazard threshold was designed as a dynamically adaptive baseline threshold, with the threshold value updated in real time to reflect previous data. The moving average of potential hazards in historical tests.

[0041] This invention introduces a reward for calculating path planning failure sensitivity; the flowchart is shown below. Figure 4 If the current environmental state is calculated hour, The tolerance constant representing the risk level is used to fine-tune the sensitivity of triggering high-risk scenario judgments. The system explicitly determines that the current test case has entered the high-risk scenario range. (Hyperbolic tangent term) The value increases rapidly, causing the dynamic weighting function to generate a gain factor greater than 1, approaching 2 at its highest, significantly amplifying the base reward; conversely, if the current area is low-risk, If the weight gain approaches 1 or even decays, then the weight gain will be close to 1.

[0042] The basic reward function is dynamically weighted to output the final reward. Using this quantitative standard, the system achieves adaptive adjustment of the exploration focus without relying on preset patterns, significantly improving the efficiency of discovering critical failure cases.

[0043] The PPO (Proximal Policy Optimization) deep reinforcement learning algorithm is used to train the test agent end-to-end. The specific process includes: S41. Initialize the policy network and value evaluation network. The network structure adopts a multi-layer fully connected neural network. The hidden layer dimension and activation function are selected according to the task complexity.

[0044] S42. Set the training hyperparameters. The specific training settings for each parameter are as follows: policy network learning rate. Value network learning rate During training, the system employs the Adam optimizer, utilizing the aforementioned learning rate to adaptively estimate the first and second moments of the network parameters, thereby achieving smooth parameter optimization in the complex state space of the testing environment; the discount factor for future rewards. Generalized dominance estimation parameters In the training loop, the system uses these two factors to perform temporal difference residual calculation on the data in the experience buffer, generating an estimate of the advantage function to quantitatively assess the contribution of the current perturbation action to the induced planning failure compared to the average policy. The policy pruning coefficients of the PPO deep reinforcement learning algorithm... During the gradient update phase, the system uses this coefficient to limit the ratio of the old and new strategies to a certain value. Internally, this prevents the test strategy from failing due to excessively large single parameter updates, ensuring that the test agent can robustly converge to the high-risk perturbation strategy.

[0045] S43. In each round of testing, the test agent generates a disturbance action based on the current policy, applies the disturbance in the simulation environment, calls the path planning algorithm under test, executes the generated flight trajectory, and determines the task result and calculates the basic reward. If the task result is to successfully induce the drone to collide or go out of bounds, a high positive reward is given; if the drone safely reaches the destination, a negative penalty is given, and the interaction sample is stored in the experience replay buffer.

[0046] S44. After accumulating a sufficient number of interaction samples, update the environment parameters according to the policy until the preset termination condition is met. The specific termination condition and execution mechanism for the network parameter policy update are as follows: Set the maximum number of interaction steps in the experience replay buffer. When the test agent accumulates interactions with the algorithm under test in the current environment to reach After this step, immediately stop exploring data collection and use this batch of sample data to analyze the parameters of the policy network and the value evaluation network. Mini-batch gradient descent updates are performed every epoch, completing one policy iteration. The testing process continues until a preset termination condition is reached, such as the maximum number of test epochs, a threshold number of epochs without discovering new failure types, or coverage saturation.

[0047] The testing agent aims to maximize the failure probability of the algorithm under test. Through real-time interaction with the path planner, it perceives the planner's responses and dynamically adjusts its testing strategy, forming a closed-loop exploration mechanism to achieve proactive testing driven by dynamic feedback. This proactive testing significantly improves vulnerability discovery efficiency and overcomes the limitations of traditional static testing, which cannot adaptively optimize.

[0048] Step S5 of the present invention includes: S51. Failure Case Storage. For each test round that triggers a failure in interactive testing, the system automatically saves the complete environment configuration, the output trajectory of the algorithm under test, the failure type label, and related metadata, stored in a structured data format, and supports visual playback. All generated failure cases include complete environment configuration, interaction trajectory, and failure reasons, possessing high reproducibility and analyzability. This provides interpretable support for algorithm iteration and security certification, and can be directly used for targeted repair, robustness enhancement, and security compliance assessment of path planning algorithms.

[0049] S52. In the initialization phase of the target new scenario, the parameters of the policy network and value network that have been trained and converged in the source scenario are directly loaded. Based on the transfer learning theory, the gradient update of the feature extraction layer in the front end of the network is frozen to retain the prior knowledge of the test agent on the general dynamics and basic spatial avoidance logic of the UAV. The network fine-tuning authority is strictly limited to the end action decision layer.

[0050] S53. To balance migration efficiency and detection cost, the sampling size of the experience replay buffer is limited during the exploration phase of the target scene, and the sample size for a single iteration of exploration of the new scene is set. Standard interaction steps only for the source scene of , serving as a benchmark for interactive exploration of the target domain.

[0051] S54. To prevent catastrophic forgetting when the network fits the features of the target domain, a small step size parameter update strategy is adopted, which involves fine-tuning the learning rate under the target scene. Set as the base learning rate of the source domain The gradient descent iteration is 0.1 times that of the target domain, and under this parameter configuration, a maximum of M rounds of restricted gradient descent iterations are performed. This mechanism ensures that the test agent can quickly capture the local adversarial features of the new environment while maximizing the reuse of the generalization optimization capability of the source domain, significantly reducing the cold start overhead of cross-scenario testing.

[0052] By introducing transfer learning, the exploration strategies learned by the test agent in one environment can be quickly transferred to a new environment without starting training from scratch. This supports efficient transfer and generalization verification across scenarios, significantly reducing testing costs and improving verification efficiency and generalization capabilities in diverse application scenarios.

[0053] This invention models the testing process as a Markov decision process and designs a specialized testing agent. This agent, interacting with the path planning algorithm under test, autonomously learns how to construct the most challenging environmental disturbances, including obstacle placement, dynamic threats, start-end point offsets, and sensor noise injection, with the goal of maximizing the algorithm's failure probability. Simultaneously, a state encoding mechanism and a reward shaping strategy are introduced to ensure that the generated test scenarios are both realistically feasible and efficiently cover diverse failure boundaries. Furthermore, this invention supports transfer learning, enabling the testing agent to quickly transfer exploration strategies learned in one scenario to new scenarios, significantly improving cross-environment testing efficiency and generalization ability.

[0054] This invention enables automated, intelligent, and high-coverage evaluation of the security, robustness, and adaptability of UAV path planning algorithms, providing reliable technical support for algorithm iteration optimization, security certification, and practical deployment. It fills the gap in the field of existing fuzzy testing technology for unmanned system decision verification and provides strong technical support for the development and deployment of highly reliable autonomous systems.

[0055] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A fuzzy testing method for UAV path planning based on reinforcement learning, characterized in that, Including the following steps: S1. Construct a perturbable three-dimensional simulation test environment and read the environmental parameters of the three-dimensional simulation test environment in real time; S2. Design a test agent, which executes state input, action output and basic reward function based on the environmental parameters; S3. Introduce a path planning failure sensitivity weighting mechanism to dynamically weight the basic reward function; S4. Perform policy training and interactive testing on the test agent, and update the environmental parameters; S5. Store the failed test cases in the interactive test and introduce a deep reinforcement learning transfer mechanism for cross-scenario transfer.

2. The fuzzy testing method for UAV path planning based on reinforcement learning according to claim 1, characterized in that, In step S1, the environmental parameters include: airspace boundaries, static obstacles, dynamic interference sources, sensing noise model, and task parameters; The perception noise model is constructed by dividing the UAV's perception state into position, velocity, and obstacle distance, and using a hybrid model of Gaussian white noise and random walk drift. Let the actual state of the drone be represented as: （1） In formula (1), This represents the true complete state vector of the tested UAV. This represents the actual three-dimensional position coordinates of the tested drone. This represents the actual distance observation vector between the tested UAV and surrounding obstacles. This represents the actual three-dimensional velocity vector of the tested UAV. The observation state after injecting noise is represented as follows: （2） In formula (2), Indicates the observation status. Indicates location observation value, Represents the observed velocity value. Indicates the observed distance to the obstacle; For location observations, inject white noise that follows a zero-mean Gaussian distribution and a drift term that accumulates over time: （3） In formula (3), Indicates location observation value, This represents the actual three-dimensional position coordinates of the tested drone. Let represent a random variable that follows a normal distribution. This represents the standard deviation of Gaussian noise at location. The set range is , Denotes the random walk drift term, satisfying , The standard deviation of Gaussian white noise represents the random increment of a single-step drift. This indicates the time difference between the tested drone and the start of the test; For velocity observations, only Gaussian white noise is injected: （4） In formula (4), Represents the observed velocity value. This represents the true three-dimensional velocity vector of the tested UAV. This represents the standard deviation of the velocity Gaussian noise. The set range is ; For obstacle distance observations, only Gaussian white noise is injected: （5） In formula (5), This represents the observed distance to the obstacle. This represents the actual distance observation vector between the tested UAV and surrounding obstacles. This represents the standard deviation of Gaussian noise at the distance to the obstacle. The set range is ; The disturbance intensity is controlled by the perceived noise model.

3. The fuzzy testing method for UAV path planning based on reinforcement learning according to claim 1, characterized in that, Step S2 includes: S21: The test agent receives the environmental configuration information for the current test round, including but not limited to the spatial coordinates of the starting point and the target point, the spatial position, quantity and geometric features of obstacles, the state of dynamic interference sources, the execution result of the algorithm under test in the previous test round and historical failure statistics. All state variables are normalized or standardized and then used as the input of the test agent. S22: The test agent outputs a set of disturbance instructions for modifying the test environment. The disturbance dimensions include spatial offset between the starting point and the target point, addition or deletion or position adjustment of static obstacles, activation of dynamic interference sources, setting of speed or trajectory parameters and adjustment of sensor noise intensity. The value range of each action component is limited to the range that ensures the physical rationality of the test scenario and the feasibility of the task. S23: Define the basic reward function based on the execution results of the path planning algorithm under test in a disturbed environment.

4. A fuzzy testing method for UAV path planning based on reinforcement learning according to claim 3, characterized in that, The basic reward function consists of three parts: a termination reward, a time penalty, and a heuristic guiding term. The calculation of the basic reward function is expressed as follows: （6） In formula (6), Represents the basic reward function. Indicates the reward at the end of the round. This indicates the single-step time penalty. This indicates a heuristic-guided reward system; The termination reward is triggered only when the drone collides, flies out of the boundary, or safely reaches the destination. If environmental disturbances cause the tested drone to collide or fly out of the safe boundary, the planning failure is triggered, and the agent is given a high positive reward. ; If the drone safely reaches the destination, a negative penalty will be imposed. ; The single-step time penalty term is a negative constant, used to encourage the agent to find the scenario that causes the drone to fail as quickly as possible. The test agent is given a small penalty for each exploration step it performs. ; The heuristic-guided reward is used to provide intensive reward feedback based on the danger level of the drone, and is based on the minimum distance between the drone's flight trajectory and the obstacle. , This indicates the minimum distance between the drone's flight path and the obstacle. This represents the adjustment coefficient. , Represents a positive minimum constant. The closer the drone gets to the obstacle, the greater the base reward the test agent receives.

5. The fuzzy testing method for UAV path planning based on reinforcement learning according to claim 1, characterized in that, Step S3 includes: S31. Define the potential hazard level of the current environment; S32. Construct a monotonically increasing dynamic weight function based on the potential hazard level; S33. Dynamically weight the basic reward function and output the weighted basic reward to guide the test agent to prioritize exploring high-risk scenarios that are more likely to expose algorithm defects.

6. The fuzzy testing method for UAV path planning based on reinforcement learning according to claim 5, characterized in that, Step S31 includes: The calculation of the potential hazard level is expressed as follows: （7） In formula (7), Indicates the time of testing The potential danger level; This represents the non-negative normalized weighting coefficient of the obstacle density term. This represents the non-negative normalized weight coefficient of the dynamic disturbance term. This represents the non-negative normalized weight coefficient of the task distance term; This indicates the number of static obstacles in the current environment; This represents a reference upper limit for the number of obstacles, used for normalization to ensure that the value of this item is within a reasonable range. Indicates the speed of motion of the dynamic interference source; The upper limit of the dynamic disturbance velocity is used for normalization and reflects the maximum disturbance intensity in a typical scenario. Represents the starting position vector of the UAV mission; This represents the target point position vector of a UAV mission; It represents a very small positive constant used to avoid zero denominators and ensure the stability of mathematical expressions.

7. The fuzzy testing method for UAV path planning based on reinforcement learning according to claim 5, characterized in that, Step S32 includes: The dynamic weighting function is calculated as follows: （8） In formula (8), Represents the dynamic weighting function. This represents the danger threshold, used to control the switching sensitivity of the weighting function. Indicates the time of testing The potential danger level.

8. The fuzzy testing method for UAV path planning based on reinforcement learning according to claim 5, characterized in that, Step S33 includes: During testing, the hazard threshold was designed as a dynamically adaptive baseline threshold, with the threshold value updated in real time to reflect previous data. The moving average of potential hazards in historical testing. If the current environmental state is calculated hour, The tolerance constant representing the risk level is used to fine-tune the sensitivity of triggering high-risk scenario judgments. The system explicitly determines that the current test case has entered the high-risk scenario range. (Hyperbolic tangent term) The value increases rapidly, causing the dynamic weighting function to generate a gain factor greater than 1, approaching 2 at its highest, significantly amplifying the base reward; conversely, if the current area is low-risk, If so, the weight gain approaches 1 or even decays; The basic reward function is dynamically weighted to output the final reward.

9. The fuzzy testing method for UAV path planning based on reinforcement learning according to claim 1, characterized in that, Step S4 includes: S41. Initialize the policy network and value evaluation network. The network structure adopts a multi-layer fully connected neural network. The hidden layer dimension and activation function are selected according to the task complexity. S42. Set training hyperparameters to prevent the test strategy from failing due to excessively large single parameter update magnitude, and ensure that the test agent can robustly converge to the high-risk perturbation strategy. S43. In each round of testing, the test agent generates a disturbance action based on the current policy, applies the disturbance in the simulation environment, calls the path planning algorithm under test, executes the generated flight trajectory, and determines the task result and calculates the basic reward. If the task result is a failure, a high positive reward is given. If the drone safely reaches the destination, a negative penalty is given. At the same time, the interaction sample is stored in the experience replay buffer. S44. After the test accumulates a sufficient number of interaction samples, the environmental parameters are updated according to the strategy until the preset termination condition is reached.

10. A fuzzy testing method for UAV path planning based on reinforcement learning according to claim 1, characterized in that, Step S5 includes: S51. For each test round that triggers failure in the interactive test, the system automatically saves the complete environment configuration, the output trajectory of the algorithm under test, the failure type label and related metadata, stores them in a structured data format, and supports visual playback. S52. In the initialization phase of the target new scenario, the parameters of the policy network and value network that have been trained and converged in the source scenario are directly loaded. The gradient update of the feature extraction layer in the front end of the network is frozen based on the transfer learning theory to retain the prior knowledge of the test agent on the general dynamics and basic spatial avoidance logic of the UAV, and the network fine-tuning authority is strictly limited to the end action decision layer. S53. Limit the sampling size of the experience replay buffer during the exploration phase of the target scene, and set the sample size for a single iteration of exploration of the new scene. Standard interaction steps only for the source scene of , serving as a benchmark for interactive exploration of the target domain; S54. Employ a small-step parameter update strategy to fine-tune the learning rate for the target scenario. Set as the base learning rate of the source domain 0.1 times the value, and perform a maximum of M rounds of restricted gradient descent iterations.

Images