Test evaluation model, method and system for path planning algorithm

Abstract

The invention discloses a test evaluation model, method and system for a path planning algorithm, and belongs to the field of algorithm test. Comprising the following steps: connecting a plurality of functions and scenes into a plurality of test scan chains; and testing the intelligent iterative optimization algorithm through each scan chain. The test scan chain comprises three packaging function chains, a packaging function test chain with noise and a metamorphic test chain based on a path planning scene, and the three packaging function chains adopt a single-mode function, a multi-mode function and a composite function for packaging. A systematic method is provided to compare and analyze the performance of different algorithms in multiple dimensions by performing comprehensive test and evaluation on multiple intelligent iterative optimization algorithms.

Description

Technical Field

[0001] The present disclosure belongs to the field of algorithm testing, and specifically relates to a test evaluation model, method and system for a path planning algorithm. Background Art

[0002] Currently, path planning algorithms have been widely used in many fields, including robotics, autonomous driving, game development, logistics systems, etc. They play an important role in improving efficiency, saving costs, optimizing resource allocation, and increasing safety. It is one of the indispensable technologies in today's society.

[0003] Common path planning algorithms now include those based on graph search, sampling, potential field, computational intelligence, etc. Among them, the intelligent iterative optimization algorithm has been widely used because of its strong global search capability, strong adaptability, and ability to handle high-dimensional problems.

[0004] However, due to the variety of intelligent iterative optimization algorithms currently available, it is difficult to determine which algorithm is optimal in practical applications, and it is difficult to distinguish which scenarios each intelligent iterative optimization algorithm is suitable for.

[0005] Based on the above research background, this patent proposes a test and evaluation method for the path planning intelligent iterative optimization algorithm, proposes a test model based on multiple scan chains, and designs a corresponding test evaluation method based on the model, which can scientifically and effectively analyze the test results, thereby realizing effective evaluation of the path planning intelligent iterative optimization algorithm. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention proposes a test evaluation model, method and system for path planning algorithms, constructs an algorithm test model based on multiple scan chains, integrates multiple functions and scenarios into the test model, simplifies and systematizes the algorithm test, and improves the test efficiency and coverage. In addition, this patent designs a comprehensive performance evaluation method based on hierarchical analysis and subjective weighting, which can scientifically and effectively analyze and evaluate the test results.

[0007] The purpose of this disclosure can be achieved through the following technical solutions:

[0008] A test and evaluation method for a path planning algorithm, the method comprising:

[0009] Connect multiple functions and scenarios into multiple test scan chains;

[0010] The intelligent iterative optimization algorithm is tested through each scan chain.

[0011] Furthermore, the test scan chain includes three encapsulation function chains, a noisy encapsulation function test chain and a path planning scenario-based degradation test chain.

[0012] Furthermore, the three encapsulation function chains are encapsulated using three types of functions: single-modal function, multi-modal function, and composite function.

[0013] The purpose of the present disclosure can also be achieved by the following technical solution: a computer-readable storage medium storing computer-executable instructions, which implement the following steps when executed:

[0014] S1: Connect multiple functions and scenarios into multiple test scan chains;

[0015] S2: Test the intelligent iterative optimization algorithm through each scan chain;

[0016] The test scan chain includes three encapsulation function chains, a noisy encapsulation function test chain and a path planning scenario-based metamorphosis test chain;

[0017] The three encapsulation function chains are encapsulated using three functions: single-mode function, multi-mode function, and composite function.

[0018] Beneficial effects of the present disclosure:

[0019] By comprehensively testing and evaluating a variety of intelligent iterative optimization algorithms, a systematic method is provided to compare and analyze the performance of different algorithms in multiple dimensions. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 is a schematic diagram of a test evaluation model based on multiple scan chains according to an embodiment of the present disclosure;

[0022] Figure 2 is a three-dimensional graph of Schffer'F6 function according to an embodiment of the present disclosure;

[0023] Figure 3 is a topographic map with a no-fly zone added in the embodiment of the present disclosure;

[0024] Figure 4 It is a two-layer degradation test framework diagram of an embodiment of the present disclosure;

[0025] Figure 5It is the optimization result of a single function test based on a single-mode encapsulation function test chain in the embodiment of the present disclosure;

[0026] Figure 6 is the test function used by the scan chain 1 of the present disclosure;

[0027] Figure 7 is the test function used by the scan chain 2 of the present disclosure;

[0028] Figure 8 is the test function used by the scan chain 3 of the present disclosure;

[0029] Fig. 9 This is a table of test times for each test scan chain disclosed in the present invention. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0031] A test and evaluation method for a path planning algorithm is presented. The method mainly includes the following two parts: one is a test and evaluation model based on multiple scan chains, and the other is a comprehensive performance evaluation using a hierarchical analysis subjective weighting method. They are introduced separately.

[0032] 1. Test evaluation model based on multiple scan chains

[0033] The present invention introduces the scan chain concept of digital integrated circuits into algorithm testing, and "connects" multiple functions and scenarios into multiple scan chains. Each scan chain tests the intelligent iterative optimization algorithm in a targeted manner, which simplifies and systematizes the testing of the intelligent iterative optimization algorithm and improves the testing efficiency and coverage.

[0034] The present invention constructs five test scan chains to form a test evaluation model based on multiple scan chains, such as Figure 1 As shown in the figure, there are three encapsulation function chains with different focuses, one encapsulation function test chain with noise, and one transformation test chain based on path planning scenarios.

[0035] 1.1 Encapsulation function test chain;

[0036] First, there are three encapsulation function test chains with different focuses, which use single-mode functions, multi-mode functions, and composite functions for encapsulation. Test chain one contains 7 single-mode functions, focusing on testing the optimization accuracy of the intelligent iterative optimization algorithm; test chain two contains 6 multi-mode functions, focusing on testing the time complexity and convergence efficiency of the intelligent iterative optimization algorithm; test chain three contains 8 composite modal functions, focusing on testing the comprehensive optimization performance of the intelligent iterative optimization algorithm. The encapsulation functions used by the three encapsulation function scanning chains are as follows: Figure 6-Figure 8 shown.

[0037] The test requirements of the algorithm under test are as follows: each algorithm under test needs to be run M times on each scan chain, and each iteration number is N times. In order to eliminate the influence of accidental results on the algorithm test results, the values ​​of the parameters M and N here need to be large enough, and the values ​​are M=30 and N=10000.

[0038] In some implementations, a combination of M=10, N=5000 or M=50, N=15000 is set.

[0039] The test results of these three test chains will be measured using the following four indicators: quantification of algorithm accuracy, quantification of algorithm time complexity, quantification of algorithm convergence efficiency, and quantification of algorithm stability.

[0040] The formula for quantifying the precision of the intelligent iterative optimization algorithm is: in a certain test function, assuming that the current best search value of the tested algorithm is fiBEST, the precision score of the algorithm in the test function is:

[0041]

[0042] The formula for quantifying the time complexity of the intelligent iterative optimization algorithm is: in a certain test function, assuming that the current best search value of the random search algorithm is fRMEAN, the running time of the random search algorithm is TRTIME, the number of times the random search algorithm calls the objective function is NRNCF, the average running time of the tested algorithm is TiTIME, the average accuracy value of the tested algorithm is fiMEAN, and the average number of times the tested function calls the objective function is NiNCF, then the time complexity score of the algorithm in the test function is

[0043]

[0044] The formula for quantifying the convergence efficiency of the intelligent iterative optimization algorithm is: in a certain test function, assuming that the current best search value of the random search algorithm is fRMEAN, the number of times the random search algorithm calls the objective function is NRNCF, the average accuracy value of the tested algorithm is fiMEAN, and the average number of times the tested function calls the objective function is NiNCF, then the convergence efficiency score of the algorithm in the test function is

[0045]

[0046] The formula for quantifying the stability of the intelligent iterative optimization algorithm is: in a certain test function, assuming that the current best search value of the random search algorithm is fRMEAN, and the accuracy standard deviation of the tested algorithm is σ iSTD , then the stability score of the algorithm in this test function is

[0047]

[0048] 1.2 Package function test chain with noise;

[0049] In order to measure the robustness of the intelligent iterative optimization algorithm, the optimization algorithm is required to be able to find a good solution when it is disturbed by external factors, and the solution should not fluctuate greatly due to the addition of interference. In the selection of the test function, it is hoped that the function can have as many feasible solution domains as possible, so as to observe the impact of noise on the test of the intelligent iterative optimization algorithm. Therefore, the improved Schffer'F6 three-dimensional function is selected, which is defined as follows:

[0050]

[0051] The function has a global optimal solution of 1 at (0,0,0). The function graph is as follows Figure 2 shown.

[0052] After the Gaussian function is introduced into the test function, it can be expressed as follows:

[0053]

[0054] Where ξ~N(μ,σ), μ represents the mean of the noise and σ represents the variance.

[0055] When the intelligent iterative optimization algorithm is used to test it, the optimal result obtained by each intelligent iterative optimization algorithm after the operation is terminated is defined as X σ , f σ (X σ ) is the observed value of the objective function, that is, the optimal value affected by noise. σ ) is the true value of the objective function, which is numerically equal to X σ The corresponding objective function value in the absence of noise. The robustness evaluation index is defined as the relative error between the observed value and the true value to measure the impact of noise on the optimization effect of the intelligent iterative optimization algorithm.

[0056]

[0057] 1.3 Transformation test chain based on path planning scenario

[0058] The task to be solved by typical path planning is to move a single or multiple agents from the starting point to the end point according to their respective established tasks in a given environment. When testing algorithms based on typical path planning scenarios, the first thing to do is to build test cases, that is, to build typical path planning scenarios. Considering the universality and scalability of the scenarios, the scenario of drone reconnaissance was selected for algorithm testing, which was mainly constructed from the following three aspects: the drone flight environment, the drone physical constraints, and the drone mission constraints.

[0059] The topography of the mountain is expressed as follows:

[0060]

[0061] Where: n represents the total number of peaks, which is 30; x i ,y i Indicates the center coordinates of the peak; h i is a terrain parameter used to control the height of the mountain; x si ,y si Control the attenuation in both directions and the slope, set them to 10 respectively.

[0062] When the drone flies along the predetermined path, it may detect detection equipment such as radar in front. At this time, the path planning system should make timely adjustments and re-plan a safe path to avoid the detection range to ensure the completion of the mission. For the sake of simplicity in calculation, the radar threat effect is treated as an equivalent terrain with the same shape and size as its range of action, and a hemispherical threat model is used to describe the relationship between the elevation data of the range of action of the radar detection equipment and the distance from the center of the radar. The radar model is established using the following formula:

[0063]

[0064] Where H(x, y) is the equivalent height of the radar threat, (x0, y0, z0) is the spatial position of the radar, and R max K is the range of radar action, h Establish relevant parameters for the radar model. Figure 3 A terrain map with no-fly zones added.

[0065] Among the physical constraints for drones, a rotorcraft drone that can take off and land vertically is selected as the experimental object. In the dynamic constraints, since the rotorcraft drone itself has inertia, it takes a certain amount of time to change its heading. Therefore, in the typical path planning scenario, the constraint of the maximum yaw angle is considered. The turning angle is calculated by the following formula.

[0066]

[0067] In the formula, l irepresents the projection of the i-th path vector on the horizontal plane, l i =(x i -x i-1 ,y i -y i-1 ) T .

[0068] For the intelligent iterative optimization algorithm test chain based on typical path planning scenarios, it is a major challenge to estimate the verification and confirmation of the entire system based on the behavior of the intelligent agent. To be precise, because it is difficult to quantify the rationality of the scheduling process and the behavior of the intelligent agent, such systems usually face the problem of test prediction. This patent solves this problem by introducing the metamorphic testing technology.

[0069] Metamorphic testing framework is a set of methodologies for test case generation and test result verification, in which metamorphic relations (MRs) are the core elements of metamorphic testing. Metamorphic transformation rules and relations are constructed based on the necessary conditions satisfied by the input and output of the object under test. Specifically, a metamorphic relation is a relation covering multiple test case inputs and outputs, i.e., MR(ω1,...,ω m ,∑(ω1),...,∑(ω m )). For any transformation relation, if we can find a test case group ω1,ω2,...,ω m Does not satisfy the relationship MR(ω1,...,ω m ,∑(ω1),...,∑(ω m )), then it can be considered that the test case group violates the transformation relationship. Because violating the transformation relationship is equivalent to violating the necessary conditions that the system must meet, we can infer that the system must contain potential defects.

[0070] This patent designs a two-layer metamorphosis test framework, which supports the construction of metamorphosis relationships through the information types contained in each layer. The two-layer metamorphosis test framework can distinguish the environment layer information and agent information required for the agent path planning problem. Based on the two-layer metamorphosis test framework, six metamorphosis relationships are proposed. The two-layer metamorphosis test framework is as follows Figure 4 shown.

[0071] In order to comprehensively consider the common characteristics of the metamorphosis test results and the intelligent iterative optimization algorithm, as well as the characteristics of typical path planning scenarios, four measurement indicators, namely effectiveness, convergence speed, path safety, and robustness, are proposed for the metamorphosis test chain based on path planning scenarios.

[0072] The effectiveness metric is to use the failure rate to represent the percentage of detected failed test cases (groups) to evaluate the performance of random testing methods and degenerate testing methods. srepresents the number of test case groups that failed the degradation test, and N represents the total number of tests.

[0073]

[0074] The convergence efficiency metric mainly comprehensively considers the optimization time and number of iterations of the intelligent iterative optimization algorithm in a typical path planning scenario. In the same typical path planning scenario, let each algorithm run independently n times, of which m times can find the optimal solution, and take the average time of m times to find the optimal solution. Average number of iterations The convergence speed metric is calculated as follows:

[0075]

[0076] The path safety metric mainly refers to measuring the safety of the path planned by the intelligent iterative optimization algorithm for typical path planning scenarios. In the UAV mountain reconnaissance scenario, we set up an objective function by integrating the turning angle constraint, maximum path constraint, extreme flight altitude constraint, and enemy radar constraint of the UAV maneuver constraints.

[0077] The turning angle constraint is calculated as follows:

[0078]

[0079] In the formula, l i represents the projection of the i-th path vector on the horizontal plane, l i =(x i -x i-1 ,y i -y i-1 ) T .

[0080] The maximum path constraint is calculated as follows:

[0081] W l =|P o P n |≤L max (14)

[0082] |P0P n | Refers to the distance between the starting point and the end point, L max Indicates the maximum flight distance.

[0083] The flight altitude constraint calculation calculates the flight altitude of the drone for each path and ensures that it does not exceed the set maximum flight altitude.

[0084] Enemy radar constraint calculation: Avoid the radar’s monitoring area based on the UAV’s planned path and the location and range of the enemy radar.

[0085] The path safety metric for UAV path planning in this project mainly considers the above four constraints and the weight coefficients of each item. After obtaining the objective function value of the intelligent iterative optimization algorithm, it is normalized to obtain the path safety metric of the intelligent iterative optimization algorithm in a typical path planning scenario.

[0086] The robustness metric is a stability metric, which is used to measure the stability of the intelligent iterative optimization algorithm in the face of extreme input conditions. In the typical path planning scenario, extreme test cases are constructed in three ways: by constructing extreme size maps, extreme obstacle maps, and extreme starting and ending positions. By measuring whether the objective function of the intelligent iterative optimization algorithm under these extreme test cases is lower than the set threshold, it is judged whether the result has successfully converged. The final intelligent iterative optimization algorithm robustness metric is obtained by statistically analyzing the proportion of problems that have been successfully optimized in all calculation test cases to the total number of running problems. The calculation formula is as follows, where N l It indicates the number of successful optimization under extreme test cases, and N indicates the total number of extreme tests.

[0087]

[0088] Now we test and verify the above method. We test seven common intelligent iterative optimization algorithms, including atomic search algorithm, artificial ecosystem optimization algorithm, particle swarm optimization algorithm, genetic algorithm, gray wolf algorithm, black widow optimization algorithm, and whale optimization algorithm, on five algorithm test scan chains. In order to make the comparison of different algorithms fair, the tests were all conducted in the same experimental environment with a 64-bit Windows 10 system and a processor of 11th Gen Intel(R) Core(TM) i9-11900K@3.50GHz3.50GHz. MATLAB programming was used to implement each test experiment.

[0089] The following are some test run results. ASO stands for atomic search algorithm, AEO stands for artificial ecosystem optimization algorithm, PSO stands for particle swarm optimization algorithm, GA stands for genetic algorithm, GWO stands for grey wolf algorithm, BWOA stands for black widow optimization algorithm, and WOA stands for whale optimization algorithm.

[0090] First, the optimization result of a single function test based on a single-mode encapsulation function test chain is presented, such as Figure 5 As shown;

[0091] Judging from the test results, the atomic search algorithm fell into a local optimum, the genetic algorithm showed a large deviation in the optimization results, and the whale optimization algorithm was at a lower-middle level in terms of convergence speed. In this test, the artificial ecosystem optimization algorithm, the particle swarm optimization algorithm, and the black widow optimization algorithm performed well.

[0092] In order to eliminate the influence of randomness on the experimental results, each test chain was tested repeatedly during the experiment. The number of repeated experiments for each test chain is as follows: Fig. 9 shown.

[0093] From the final test results, the genetic algorithm has poor convergence performance and good path security performance; the atomic search algorithm has poor effectiveness performance, medium stability performance, good convergence performance, and poor path security performance; the particle swarm algorithm performs well in all aspects; the whale optimization algorithm has good effectiveness performance, good convergence performance, poor stability and path security performance; the gray wolf optimization algorithm has very poor path security performance and average performance in other aspects; the artificial ecology optimization algorithm has good convergence efficiency performance and good path security performance; the black widow optimization algorithm has good path security performance. In summary, the particle swarm algorithm and the black widow optimization algorithm perform well in all aspects, while the atomic search algorithm and the gray wolf optimization algorithm perform poorly in this algorithm test based on multi-scan chains.

[0094] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0095] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements all fall within the scope of the present invention to be protected.

Claims

1. A test and evaluation method for a path planning algorithm, characterized in that: The method comprises: Connect multiple functions and scenarios into multiple test scan chains; The intelligent iterative optimization algorithm is tested through each scan chain.

2. A test and evaluation method for a path planning algorithm according to claim 1, characterized in that: The test scan chain includes three encapsulation function chains, a noisy encapsulation function test chain and a path planning scenario-based degradation test chain.

3. A test and evaluation method for a path planning algorithm according to claim 1, characterized in that: The three encapsulation function chains are encapsulated using three functions: single-mode function, multi-mode function, and composite function.

4. A test and evaluation method for a path planning algorithm according to claim 1, characterized in that: The specific method of testing the intelligent iterative optimization algorithm is as follows; The algorithm under test is run 20-40 times on each scan chain, with 5000-15000 iterations each time, to eliminate the influence of accidental results on the algorithm test results.

5. A test and evaluation method for a path planning algorithm according to claim 1, characterized in that: The test results of the encapsulated function chain are measured using four indicators: accuracy quantification, time complexity quantification, convergence efficiency quantification, and stability quantification.

6. A test and evaluation method for a path planning algorithm according to claim 1, characterized in that; The package function test chain with noise selects the improved Schaffer's F6 three-dimensional function with multiple feasible solution domains as the test function, and introduces Gaussian noise into the test function so that the intelligent iterative optimization algorithm can test it; By defining the optimal result obtained by the intelligent iterative optimization algorithm after the operation is terminated as observation; Compare the observed value with the true value of the objective function in the noise-free case; The optimization effect of the intelligent iterative optimization algorithm under noise interference is measured by calculating the relative error between the observed value and the true value to evaluate the robustness of the algorithm.

7. A test and evaluation method for a path planning algorithm according to claim 1, characterized in that; The test methods of the metamorphosis test chain based on the path planning scenario include: Build typical path planning scenarios to test intelligent iterative optimization algorithms; In the constructed scenario, the UAV flight environment, physical constraints, and mission constraints are considered; Use mountain terrain models and radar threat models to simulate obstacles and threats in the environment; Generate test cases through the metamorphosis test framework and verify the performance of the intelligent iterative optimization algorithm; Design transformation relations as the relations between test case input and output to evaluate the rationality of agent behavior and the integrity of the system; During the testing process, if it is found that the test case group does not satisfy the transformation relationship, it is inferred that there are potential defects in the system.

8. A test and evaluation method for a path planning algorithm according to claim 7, characterized in that; The design of metamorphosis relationship adopts a two-layer metamorphosis test framework, which supports the construction of metamorphosis relationship through the information type of each level. The two-layer metamorphosis test framework is the environment layer information and the intelligent agent information.

9. A test and evaluation method for a path planning algorithm according to claim 8, characterized in that: The test of the path planning scenario-based metamorphosis test chain includes the following metrics: The effectiveness metric uses the failure rate to represent the percentage of failed test cases detected. Convergence efficiency measurement index, which comprehensively considers the optimization time and number of iterations of the intelligent iterative optimization algorithm in typical path planning scenarios; Path safety metrics measure the safety of the path planned by the intelligent iterative optimization algorithm, including turning angle constraints, maximum path constraints, extreme flight altitude constraints, and enemy radar constraints; Robustness metric that measures the stability of intelligent iterative optimization algorithms in the face of extreme input conditions.

10. A computer-readable storage medium storing computer-executable instructions, which when executed implement the following steps: S1: Connect multiple functions and scenarios into multiple test scan chains; S2: Test the intelligent iterative optimization algorithm through each scan chain; The test scan chain includes three encapsulation function chains, a noisy encapsulation function test chain and a path planning scenario-based metamorphosis test chain; The three encapsulation function chains are encapsulated using three functions: single-mode function, multi-mode function, and composite function.

Images