A test and evaluation model, method, and system for path planning algorithms.

Abstract

This invention discloses a testing and evaluation model, method, and system for path planning algorithms, belonging to the field of algorithm testing. It includes connecting multiple functions and scenarios into multiple test scan chains; and testing intelligent iterative optimization algorithms through each scan chain. The test scan chains include three encapsulated function chains, one encapsulated function test chain with noise, and one metamorphic test chain based on the path planning scenario. The three encapsulated function chains use single-modal functions, multi-modal functions, and composite functions for encapsulation. By comprehensively testing and evaluating various intelligent iterative optimization algorithms, a systematic method is provided to compare and analyze the performance of different algorithms across multiple dimensions.

Description

Technical Field

[0001] This disclosure pertains to the field of algorithm testing, specifically relating to a testing and evaluation model, method, and system for path planning algorithms. Background Technology

[0002] Currently, path planning algorithms have been widely used in many fields, including robotics, autonomous driving, game development, and logistics systems. They play an important role in improving efficiency, saving costs, optimizing resource allocation, and increasing safety, and are one of the indispensable technologies in modern society.

[0003] Common path planning algorithms include graph search, sampling, potential field, and computational intelligence algorithms. Among them, intelligent iterative optimization algorithms have been widely used due to their strong global search capabilities, high adaptability, and ability to handle high-dimensional problems.

[0004] However, due to the variety of intelligent iterative optimization algorithms currently available, it is difficult to determine which algorithm is optimal in practical applications, and it is also difficult to distinguish which intelligent iterative optimization algorithm is suitable for which scenario.

[0005] Based on the above research background, this patent proposes a test and evaluation method for intelligent iterative optimization algorithms for path planning. It proposes a test model based on multi-scan chains and designs a corresponding test and evaluation method based on the model. This method can scientifically and effectively analyze the test results, thereby achieving an effective evaluation of intelligent iterative optimization algorithms for path planning. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention proposes a testing and evaluation model, method, and system for path planning algorithms. It constructs an algorithm testing model based on multi-scan chains, integrating multiple functions and scenarios into the testing model, simplifying and systematizing algorithm testing, and improving testing efficiency and coverage. Furthermore, this patent designs a comprehensive performance evaluation method based on hierarchical analysis and subjective weighting, enabling scientific and effective analysis and evaluation of test results.

[0007] The objective of this disclosure can be achieved through the following technical solutions:

[0008] A method for testing and evaluating path planning algorithms, the method comprising:

[0009] Connect multiple functions and scenarios into multiple test scan chains;

[0010] The intelligent iterative optimization algorithm was tested on each scan chain.

[0011] Furthermore, the test scan chain includes three encapsulation function chains, one noisy encapsulation function test chain, and one metamorphosis test chain based on path planning scenarios.

[0012] Furthermore, the three encapsulation function chains are encapsulated using three types of functions: single-modal functions, multi-modal functions, and composite functions.

[0013] The objective of this disclosure can also be achieved through the following technical solution: a computer-readable storage medium storing computer-executable instructions, which, when executed, perform the following steps:

[0014] S1: Connect multiple functions and scenarios into multiple test scan chains;

[0015] S2: Test the intelligent iterative optimization algorithm through each scan chain;

[0016] The test scan chain includes three encapsulation function chains, one noisy encapsulation function test chain, and one metamorphosis test chain based on path planning scenarios.

[0017] The three encapsulation function chains are encapsulated using three types of functions: single-modal functions, multi-modal functions, and composite functions.

[0018] The beneficial effects of this disclosure are:

[0019] By conducting comprehensive testing and evaluation of various intelligent iterative optimization algorithms, a systematic approach is provided to compare and analyze the performance of different algorithms across multiple dimensions. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of a test and evaluation model based on a multi-scan chain according to an embodiment of this disclosure;

[0022] Figure 2 This is a three-dimensional graph of the Schffer'F6 function in an embodiment of this disclosure;

[0023] Figure 3 This is a topographic map showing the addition of no-fly zones in an embodiment of this disclosure;

[0024] Figure 4 This is a two-layer metamorphosis test framework diagram of an embodiment of this disclosure;

[0025] Figure 5This is the result of single-function test optimization based on a single-modal encapsulated function test chain in the embodiments of this disclosure;

[0026] Figure 6 This is the test function used in this public scan chain;

[0027] Figure 7 This is the test function used in this public scan chain 2;

[0028] Figure 8 This is the test function used in this public scan chain 3;

[0029] Figure 9 This is a table of the number of tests for each test scan chain disclosed in this publication. Detailed Implementation

[0030] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0031] A test and evaluation method for path planning algorithms is presented, which mainly consists of two parts: a test and evaluation model based on multi-scan chains and a comprehensive performance evaluation using the analytic hierarchy process (AHP) subjective weighting method. These will be introduced separately.

[0032] 1. Test and evaluation model based on multi-scan chain

[0033] This invention introduces the concept of scan chains from digital integrated circuits into algorithm testing, connecting multiple functions and scenarios into multiple scan chains. Each scan chain is specifically designed to test intelligent iterative optimization algorithms, which simplifies and systematizes the testing of intelligent iterative optimization algorithms, improving testing efficiency and coverage.

[0034] This invention constructs five test scan chains, forming a test evaluation model based on multi-scan chains, such as... Figure 1 As shown. This includes three encapsulation function chains, each with its own focus, a noisy encapsulation function test chain, and a metamorphosis test chain based on a path planning scenario.

[0035] 1.1 Encapsulate the function test chain;

[0036] First, there are three encapsulation function test chains, each with its own focus, using single-modal functions, multi-modal functions, and composite functions for encapsulation. Test chain one contains 7 single-modal functions, focusing on testing the optimization accuracy of the intelligent iterative optimization algorithm; test chain two contains 6 multi-modal functions, focusing on testing the time complexity and convergence efficiency of the intelligent iterative optimization algorithm; and test chain three contains 8 composite modal functions, focusing on testing the overall optimization performance of the intelligent iterative optimization algorithm. The encapsulation functions used in the three encapsulation function scan chains are as follows: Figures 6-8 As shown.

[0037] The testing requirements for the algorithms under test are as follows: each algorithm under test needs to be run M times on each scan chain, with N iterations per iteration. In order to eliminate the influence of random results on the algorithm test results, the values ​​of parameters M and N need to be large enough, with values ​​of M=30 and N=10000.

[0038] In some implementations, M=10, N=5000 or a combination of M=50, N=15000 is used.

[0039] The test results of these three test chains will be measured using the following four metrics: algorithm accuracy quantification, algorithm time complexity quantification, algorithm convergence efficiency quantification, and algorithm stability quantification.

[0040] The formula for quantifying the accuracy of intelligent iterative optimization algorithms is as follows: In a certain test function, assuming the current best search value of the algorithm under test is fiBEST, then the accuracy score of the algorithm in that test function is:

[0041]

[0042] The formula for quantifying the time complexity of intelligent iterative optimization algorithms is as follows: In a certain test function, assuming the current optimal search value of the random search algorithm is fRMEAN, the running time of the random search algorithm is TRTIME, the number of times the random search algorithm calls the objective function is NRNCF, the average running time of the tested algorithm is TiTIME, the average accuracy of the tested algorithm is fiMEAN, and the average number of times the tested function calls the objective function is NiNCF, then the time complexity score of the algorithm in this test function is:

[0043]

[0044] The formula for quantifying the convergence efficiency of the intelligent iterative optimization algorithm is as follows: In a certain test function, assuming the current best search value of the random search algorithm is fRMEAN, the number of times the random search algorithm calls the objective function is NRNCF, the average precision of the tested algorithm is fiMEAN, and the average number of times the tested function calls the objective function is NiNCF, then the convergence efficiency score of the algorithm in this test function is:

[0045]

[0046] The formula for quantifying the stability of intelligent iterative optimization algorithms is as follows: In a certain test function, assuming the current optimal search value of the random search algorithm is fRMEAN, and the standard deviation of the accuracy of the algorithm under test is σ... iSTD The stability score of the algorithm in this test function is then...

[0047]

[0048] 1.2 Test chains for encapsulated functions with noise;

[0049] To measure the robustness of the intelligent iterative optimization algorithm—that is, the algorithm should still be able to find a good solution when subjected to external disturbances, without significant fluctuations in the solution due to the introduction of disturbances—the test function should ideally have as many feasible solution domains as possible to observe the impact of noise on the intelligent iterative optimization algorithm test. Therefore, the improved Schffer'F6 three-dimensional function was selected, defined as follows:

[0050]

[0051] The function has a global optimum of 1 at (0,0,0). The function graph is shown below. Figure 2 As shown.

[0052] The test function, after incorporating the Gaussian function, can be expressed as follows:

[0053]

[0054] Where ξ ~ N(μ,σ), where μ represents the mean of the noise and σ represents the variance.

[0055] When testing the intelligent iterative optimization algorithm, the optimal result obtained by each intelligent iterative optimization algorithm after termination is defined as X. σ f σ (X σ f(X) represents the observed value of the objective function, i.e., the optimal value affected by noise. σ ) represents the true value of the objective function, that is, numerically equal to X. σ The objective function value under noise-free conditions. The robustness evaluation index is defined as the relative error between the observed value and the true value to measure the impact of noise on the optimization performance of the intelligent iterative optimization algorithm.

[0056]

[0057] 1.3 Transformation Test Chain Based on Path Planning Scenarios

[0058] Typical path planning aims to solve the problem of moving one or more agents from a starting point to a destination within a given environment, according to their respective predetermined tasks. When testing algorithms based on typical path planning scenarios, the first step is to construct test cases, i.e., to build typical path planning scenarios. Considering the universality and scalability of the scenario, a scenario involving UAV reconnaissance was chosen for algorithm testing. This scenario was constructed primarily from three aspects: the UAV flight environment, the UAV physical constraints, and the UAV task constraints.

[0059] The topography of a mountain peak is represented by the following formula:

[0060]

[0061] In the formula: n represents the total number of mountain peaks, which is taken as 30; x i y i h represents the center coordinates of the mountain peak; i These are terrain parameters used to control the height of mountain peaks; x si y si Control the attenuation in both directions and the slope, each set to 10.

[0062] When a drone flies along a predetermined path, it may detect radar or other detection equipment ahead. In this case, the path planning system should adjust promptly and replan a safe path to avoid the detection range, ensuring mission completion. For computational simplicity, the radar threat effect is treated as equivalent terrain of the same shape and size as its effective range. A hemispherical threat model is used to describe the relationship between the elevation data of the radar detection equipment's effective range and the distance from the radar center. The radar model is represented by the following formula:

[0063]

[0064] In the formula, H(x, y) is the radar threat equivalent elevation, (x0, y0, z0) is the spatial position of the radar, and R max K represents the range of radar coverage. h Establish relevant parameters for the radar model. Figure 3 It's a topographic map with no-fly zones added.

[0065] In the physical constraints of the drone, a rotary-wing drone capable of vertical takeoff and landing was selected as the experimental subject. In the dynamic constraints, since the rotary-wing drone itself has inertia, changing its course requires a certain amount of time; therefore, in a typical path planning scenario, the constraint of the maximum yaw angle is considered. The turning angle is calculated using the following formula.

[0066]

[0067] In the formula, l iLet l represent the projection of the i-th path vector onto the horizontal plane. i =(x i -x i-1 y i -y i-1 ) T .

[0068] For test chains of intelligent iterative optimization algorithms based on typical path planning scenarios, verifying and validating the entire system by estimating the behavior of the agent is a significant challenge. More precisely, because it is difficult to quantify the rationality of the scheduling process and agent behavior, such systems typically face the test oracle problem. This patent addresses this problem by introducing metamorphic testing technology.

[0069] Metamorphic testing framework is a methodology for test case generation and test result verification. Metamorphic Relations (MRs) are the core element of metamorphic testing, constructed based on the necessary conditions satisfied by the inputs and outputs of the tested object to establish metamorphic transformation rules and relationships. Specifically, a metamorphic relation is a relationship encompassing multiple test case inputs and outputs, i.e., MR(ω1,...,ω...). m ,∑(ω1),...,∑(ω m For any metamorphic relation, if we can find a test case set ω1,ω2,...,ω m The relation MR(ω1,...,ω) is not satisfied. m ,∑(ω1),...,∑(ω m If the test case group violates the metamorphic relation, then we can conclude that it violates the necessary conditions that the system must satisfy. Because violating the metamorphic relation is equivalent to violating the necessary conditions that the system must satisfy, we can infer that the system necessarily contains a potential defect.

[0070] This patent designs a two-layer metamorphic testing framework. The information types inherent in each layer support the construction of metamorphic relationships. This two-layer framework can distinguish between the environmental information and agent information required for agent path planning problems. Based on this framework, six metamorphic relationships are proposed. The two-layer metamorphic testing framework is as follows: Figure 4 As shown.

[0071] To comprehensively consider the common characteristics of the transformation test results and the intelligent iterative optimization algorithm, as well as the characteristics of typical path planning scenarios, four metrics are proposed for the transformation test chain based on path planning scenarios: effectiveness, convergence speed, path safety, and robustness.

[0072] Effectiveness metrics use failure rate to represent the percentage of failed test cases (groups) out of the total, used to evaluate the performance of randomized testing methods and metamorphic testing methods. sN represents the number of test case groups for the metamorphosis test failure, and N represents the total number of tests.

[0073]

[0074] Convergence efficiency metrics primarily consider the optimization time and number of iterations of intelligent iterative optimization algorithms in typical path planning scenarios. In the same typical path planning scenario, each algorithm is run independently n times, and the optimal solution is found in m of those n runs. The average time taken from these m runs is then used to calculate the optimal solution. Average number of iterations The convergence speed metric is calculated as follows:

[0075]

[0076] The path safety metric primarily measures the safety of paths planned by intelligent iterative optimization algorithms in typical path planning scenarios. In the UAV mountain reconnaissance scenario, we established an objective function that integrates UAV maneuver constraints such as turning angle constraints, maximum path constraints, extreme flight altitude constraints, and enemy radar constraints.

[0077] The turning angle constraint is calculated as follows:

[0078]

[0079] In the formula, l i Let l represent the projection of the i-th path vector onto the horizontal plane. i =(x i -x i-1 y i -y i-1 ) T .

[0080] The maximum path constraint is calculated as follows:

[0081] W l =|P o P n |≤L max (14)

[0082] |P0P n | refers to the distance between the starting point and the ending point, L max This indicates the maximum flight distance.

[0083] The extreme flight altitude constraint calculation calculates the drone's flight altitude for each path segment and ensures that it does not exceed the set maximum flight altitude.

[0084] Enemy radar constraint calculation: Based on the UAV's planned path and the location and range of enemy radar, avoid the radar's monitoring area.

[0085] This research project focuses on the path safety metrics for UAV path planning, primarily considering the four constraints mentioned above and their respective weighting coefficients. After obtaining the objective function value of the intelligent iterative optimization algorithm, normalization is performed to obtain the path safety metrics for the intelligent iterative optimization algorithm in typical path planning scenarios.

[0086] Robustness metrics, also known as stability metrics, are used to measure the stability of intelligent iterative optimization algorithms under extreme input conditions. In a typical path planning scenario, extreme test cases are constructed using three methods: extreme map size, extreme obstacle map, and extreme start and end point locations. The convergence of the intelligent iterative optimization algorithm is determined by whether its objective function falls below a set threshold under these extreme test cases. The final robustness metric for the intelligent iterative optimization algorithm is obtained by statistically analyzing the proportion of problems successfully optimized across all test cases out of the total number of runs. The calculation formula is as follows, where N... l This represents the number of times optimization was successfully achieved under extreme test cases, where N represents the total number of extreme tests.

[0087]

[0088] The above methods are now tested and verified. Seven common intelligent iterative optimization algorithms—atomic search algorithm, artificial ecosystem optimization algorithm, particle swarm optimization algorithm, genetic algorithm, gray wolf algorithm, black widow optimization algorithm, and whale optimization algorithm—are tested on five algorithm test scan chains. To ensure fairness in the comparison of different algorithms, all tests are conducted under the same experimental environment: a 64-bit Windows 10 system with an 11th Gen Intel(R) Core(TM) i9-11900K processor at 3.50GHz. MATLAB programming is used to implement each test experiment.

[0089] The following are some test results. ASO represents the Atomic Search Algorithm, AEO represents the Artificial Ecosystem Optimization Algorithm, PSO represents the Particle Swarm Optimization Algorithm, GA represents the Genetic Algorithm, GWO represents the Gray Wolf Algorithm, BWOA represents the Black Widow Algorithm, and WOA represents the Whale Algorithm.

[0090] First, the optimization results of single-function tests based on the single-modal encapsulation function test chain are obtained, such as... Figure 5 As shown;

[0091] Based on the test results, the Atomic Search algorithm got stuck in a local optimum, the Genetic Algorithm showed a significant deviation in its optimization results, and the Whale Optimization Algorithm had a below-average convergence speed. In this test, the Artificial Ecosystem Optimization Algorithm, Particle Swarm Optimization Algorithm, and Black Widow Optimization Algorithm performed well.

[0092] To eliminate the influence of randomness on the experimental results, each test chain was repeatedly tested during the experiment. The number of repetitions for each test chain was as follows: Figure 9 As shown.

[0093] Based on the final test results, the genetic algorithm performed poorly in convergence but excellently in path safety; the atomic search algorithm performed poorly in efficiency, moderately in stability, and well in convergence but poorly in path safety; the particle swarm optimization algorithm performed well in all aspects; the whale optimization algorithm performed well in efficiency and convergence, but poorly in stability and path safety; the gray wolf optimization algorithm performed very poorly in path safety and averagely in other aspects; the artificial ecology optimization algorithm performed well in convergence efficiency and path safety; and the black widow optimization algorithm performed well in path safety. In summary, the particle swarm optimization algorithm and the black widow optimization algorithm performed well in all aspects, while the atomic search algorithm and the gray wolf optimization algorithm performed poorly in this multi-scan chain-based algorithm test.

[0094] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0095] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.

Claims

1. A method for testing and evaluating path planning algorithms, characterized in that, The method includes: Connect multiple functions and scenarios into multiple test scan chains; The intelligent iterative optimization algorithm was tested on each scan chain; The test scan chain includes three encapsulation function chains, one noisy encapsulation function test chain, and one metamorphosis test chain based on path planning scenarios. The three encapsulation function chains are encapsulated using three types of functions: single-modal functions, multi-modal functions, and composite functions. The test results of the encapsulated function chain are measured using four metrics: precision quantization, time complexity quantization, convergence efficiency quantization, and stability quantization. The test chain for the encapsulated function with noise selects the improved Schaffer's F6 three-dimensional function with multiple feasible solution domains as the test function, and introduces Gaussian noise into the test function to enable the intelligent iterative optimization algorithm to test it. The optimal result obtained by the intelligent iterative optimization algorithm after its execution is terminated is defined as the observation. Compare the observed values ​​with the true values ​​of the objective function under noise-free conditions; The optimization performance of the intelligent iterative optimization algorithm under noise interference is measured by calculating the relative error between the observed value and the true value, so as to evaluate the robustness of the algorithm. The testing methods for metamorphosis test chains based on path planning scenarios include: Construct typical path planning scenarios to test the intelligent iterative optimization algorithm; In the constructed scenario, the drone's flight environment, physical constraints, and mission constraints are considered; Mountain terrain models and radar threat models are used to simulate obstacles and threats in the environment; Test cases are generated using the metamorphic testing framework, and the performance of the intelligent iterative optimization algorithm is verified. Design metamorphic relationships as input and output relationships for test cases to evaluate the rationality of agent behavior and the integrity of the system; If, during testing, a test case group is found to not satisfy the metamorphic relationship, it is inferred that there is a potential defect in the system.

2. The method for testing and evaluating path planning algorithms according to claim 1, characterized in that, The specific testing method for the intelligent iterative optimization algorithm is as follows; The algorithm under test is run 20-40 times on each scan chain, with 5000-15000 iterations each time, to eliminate the influence of random results on the algorithm test results.

3. The testing and evaluation method for path planning algorithms according to claim 2, characterized in that... ； The design of metamorphic relationships adopts a two-layer metamorphic testing framework, which supports the construction of metamorphic relationships through the information types contained in each layer. The two layers of the metamorphic testing framework are environmental layer information and agent information.

4. The testing and evaluation method for path planning algorithms according to claim 3, characterized in that, The testing of the metamorphosis test chain based on the path planning scenario includes the following metrics: A validity metric, using the failure rate to represent the percentage of test cases that detected failures out of the total; Convergence efficiency metrics comprehensively consider the optimization time and number of iterations of intelligent iterative optimization algorithms in typical path planning scenarios. Path safety metrics measure the safety of paths planned by intelligent iterative optimization algorithms, including turning angle constraints, maximum path constraints, extreme flight altitude constraints, and enemy radar constraints. Robustness metrics measure the stability of intelligent iterative optimization algorithms in the face of extreme input conditions.

5. A computer-readable storage medium storing computer-executable instructions for implementing a test and evaluation method for a path planning algorithm as described in any one of claims 1-4.

Images