Autonomous navigation system robustness evaluation method

Abstract

The invention relates to an autonomous navigation system robustness evaluation method, in order to realize accurate, comprehensive and quantitative evaluation of the robustness of an autonomous navigation system and further provide a basis and a direction for research and improvement of an autonomous navigation technology, the invention summarizes and concludes robustness evaluation indexes of the autonomous navigation system. An index calculation method based on comparison of target values before and after interference application and comparison of actual values before and after interference application is established; and carrying out index fusion on the basis to obtain an evaluation score, so as to realize the evaluation of the robustness of autonomous navigation. The method is suitable for comprehensive evaluation of robustness under different scenes and different tasks, the selected indexes can comprehensively describe the navigation performance of the autonomous navigation system in time and space, and an evaluation result can be quantitatively given. The invention belongs to the field of autonomous navigation system evaluation technology research.

Description

Technical Field

[0001] The invention relates to a method for evaluating the robustness of an autonomous navigation system, and belongs to the research field of autonomous navigation system evaluation technology. Background Art

[0002] An ideal unmanned system should have the ability to sense the external environment with the help of sensors without external interference, and thus move autonomously to the target area. Autonomous navigation technology is the core technology to achieve this goal. Intelligent entities with autonomous navigation capabilities can be collectively referred to as autonomous navigation systems, such as autonomous mobile robots, autonomous unmanned submarines, etc.

[0003] Robustness refers to the ability of a system to withstand failures and interferences, that is, whether the system can maintain stable operation or return to a stable state when facing various interferences and uncertainties. Since the working environment of autonomous navigation systems is often complex and changeable, and the scope of tasks performed is relatively wide, when facing environmental interference and uncertainties, it is also necessary to keep the system performance as stable as possible to complete the specified tasks, so higher requirements are placed on the robustness of the system. In summary, it is very important to conduct an accurate and reasonable performance evaluation of the robustness of autonomous navigation systems.

[0004] However, the current research on robustness evaluation of autonomous navigation systems mainly has the following problems:

[0005] (1) Such as quantitative description of external environment changes and interference to the autonomous navigation system;

[0006] (2) How to quantitatively calculate the evaluation indicators that reflect the robustness of autonomous navigation;

[0007] (3) How to integrate various evaluation indicators to form the final robustness evaluation result.

[0008] Most of the existing performance evaluation methods evaluate static targets and calculate various evaluation indicators by measuring their physical information. However, as a mobile target, the autonomous navigation system will generate a large amount of data in time and space. To comprehensively evaluate its robustness, it is necessary to reasonably utilize the path and time data generated during the autonomous navigation process, which puts forward new requirements for the calculation of evaluation indicators and the construction of the evaluation system. Summary of the invention

[0009] In order to solve the above problems existing in the prior art, the present invention further proposes a method for evaluating the robustness of an autonomous navigation system.

[0010] The technical solution adopted by the present invention to solve the above problems is:

[0011] The present invention comprises the following steps:

[0012] Step 1: Global path planning in a specified original scene obtains the expected target value and the actual value of autonomous navigation;

[0013] Step 2: By adding interference and noise, a scene after adding interference is generated, and on this basis, global path planning is performed to obtain the target value after adding interference and the actual value of autonomous navigation;

[0014] Step 3, compare the target value before adding the interference with the target value after adding the interference, and calculate the corresponding value; then compare the actual value before adding the interference with the actual value after adding the interference, and calculate the corresponding value;

[0015] Step 4: Compare the two differences obtained in step 3 to obtain an evaluation score.

[0016] Furthermore, the indicators based on the comparison between the target value before and after the interference and the comparison between the actual value before and after the interference are calculated by interference loss, L1 and L2 loss functions, Pearson correlation coefficient and coefficient of determination COD.

[0017] Furthermore, the interference loss includes the consumption of time, path and energy, the time consumption change is the change of the time consumption after the interference is added and the time consumption before the interference is added, the path consumption change is the change of the path consumption after the interference is added and the path consumption before the interference is added, and the energy consumption change refers to the change of the energy consumption after the interference is added and the energy consumption before the interference is added; the calculation is as follows:

[0018] The change in consumption after the interference is added is divided by the value before the interference is added to get the change rate, and then the change rates are summed.

[0019]

[0020] Where ΔT is the change in time consumption after the interference is added; E T is the change in time consumption before interference is added; ΔS is the change in path consumption after interference is added; E S is the change in path consumption before interference is added; ΔE is the change in energy consumption after interference is added; E E is the change in energy consumption before interference is added.

[0021] Furthermore, the L1 and L2 loss functions are calculated as follows: i and f(x i ) are the ordinates of the theoretical path and the actual path respectively: the difference between the actual values ​​of autonomous navigation before and after adding noise and interference and without adding noise and interference is calculated by L1 norm loss function and L2 norm loss function;

[0022]

[0023] The same principle as using the L1 norm loss function and the L2 norm loss function in generalization evaluation is used in the robustness evaluation of autonomous navigation. If the difference between the target values ​​before and after adding noise is large, the L1 norm should be selected for calculation, otherwise the L2 norm should be selected for calculation.

[0024] Furthermore, the Pearson correlation coefficient, r, has a larger absolute value, indicating a stronger correlation, which can be used to judge the quality of trajectory overlap; the value range of r is [-1,1], r>0 indicates a positive correlation, r<0 indicates a negative correlation, and r=0 indicates a non-linear relationship;

[0025]

[0026] The overall correlation coefficient r is defined as the ratio of the covariance between two trajectories X and Y to the product of their standard deviations.

[0027] Furthermore, the coefficient of determination is calculated as follows:

[0028] First, there is a set of sample data sets (x1, y1), (x2, y2), ... (x n ,y n ), the predicted value is obtained by model calculation Observed data mean Then substitute it into the following formula to get RSS and TSS, and then get the coefficient of determination R;

[0029]

[0030] TSS is the total sum of squares, RSS is the residual sum of squares. The larger the value of the coefficient of determination, the closer the observed value is to the calculated value, which means that the anti-interference ability is stronger.

[0031] Furthermore, the above four indicators are weighted and integrated to obtain the robustness capability A of autonomous navigation. robust , as shown in formula (6), r is the Pearson correlation coefficient, is the inverse of the interference loss, 1, R is the L1 / L2 norm loss function 2 is the coefficient of determination;

[0032]

[0033] The A obtained by using the percentage system robust The value is scored, and the score is used to indicate the generalization quality;

[0034] Compare the target value before adding the interference with the target value after adding the interference and calculate the corresponding value Arobust1 , and then compare the actual value before adding the interference with the actual value after adding the interference, and calculate the corresponding value A robust2 , then A robust1 and A robust2 For comparison, the robustness scoring formula is shown in formula (7):

[0035]

[0036] If the score is closer to 100, that is, the closer the robustness of the actual system is to the robustness calculated by the target value, the better the system's autonomous navigation robustness is. If the score exceeds 100, it means that the interference of the scene is unreasonable and needs to be re-evaluated.

[0037] The beneficial effects of the present invention are:

[0038] 1. The present invention summarizes the robustness evaluation indicators of the autonomous navigation system, establishes an indicator calculation method based on the comparison of target values ​​before and after interference and the comparison of actual values ​​before and after interference, and on this basis, performs indicator fusion to obtain the robustness evaluation result of the autonomous navigation system.

[0039] 2. The autonomous navigation system robustness evaluation method proposed in the present invention is suitable for comprehensive robustness evaluation in different scenarios and tasks. The selected indicators can comprehensively describe the navigation performance of the autonomous navigation system in time and space, and can quantitatively give the evaluation results.

[0040] 3. The present invention realizes an accurate and comprehensive evaluation of the robustness of the autonomous navigation system, thereby providing a basis and direction for the research and improvement of autonomous navigation technology. At the same time, the present invention also has certain reference significance for performance evaluation research in other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is the overall flow chart of the present invention;

[0042] Figure 2 It is a schematic diagram of the robustness evaluation index design scheme of the present invention;

[0043] Figure 3 It is a schematic diagram of the fusion of robustness indicators of the autonomous navigation system of the present invention. DETAILED DESCRIPTION

[0044] The autonomous navigation system robustness evaluation method described in this embodiment includes the following steps:

[0045] Step 1: Global path planning in the specified original scene obtains the expected target value and the actual value of autonomous navigation;

[0046] The target value of the autonomous navigation system through capability assessment is the expected target value obtained by the global path planning algorithm under the specified scenario, including path point set, path length, path energy consumption, path time consumption and other data. Taking the unmanned vehicle as an example, since the A* algorithm has the characteristic of obtaining the optimal path, the A* algorithm is used for path planning and obtains the target path point set. Then, based on the target path point set, the target value data such as path length, path energy consumption, and path time consumption are calculated.

[0047] Step 2: By adding interference and noise, a scene after adding interference is generated, and then global path planning is performed on this basis to obtain the target value after adding interference and the actual value of autonomous navigation;

[0048] Step 3, compare the target value before adding the interference with the target value after adding the interference, and calculate the corresponding value; then compare the actual value before adding the interference with the actual value after adding the interference, and calculate the corresponding value;

[0049] According to the data characteristics of the autonomous navigation system path point set, path length, path energy consumption, path time consumption, etc., the robustness evaluation index can be calculated. On this basis, the calculation results of the target value index and the actual value index calculation results are compared to obtain the robustness evaluation results. Therefore, appropriate technical indicators should be selected. Figure 2 As shown in the figure, the technical indicator design scheme of the robustness evaluation method is formulated. According to the relationship between the target value and the target value, and the actual value and the actual value after the interference is added, four different evaluation indicator calculation rules are formulated, namely interference loss, L1 and L2 loss function, Pearson correlation coefficient, and coefficient of determination COD, which will be introduced respectively below.

[0050] 3.1 Interference loss

[0051] Interference Consume is defined as the time, path, and energy losses caused by the introduction of interference. The change in time consumption refers to the change in time consumption after the interference is added compared to the time consumption before the interference is added. The change in path consumption refers to the change in path consumption after the interference is added compared to the path consumption before the interference is added. The change in energy consumption refers to the change in energy consumption after the interference is added compared to the energy consumption before the interference is added.

[0052] Considering that the change in path length, time and energy consumption are of different dimensions, the change needs to be dimensionless, that is, the change in consumption after the interference is added is divided by the value before the interference is added to obtain the change rate, and then the change rates are summed.

[0053]

[0054] 3.2 L1 and L2 loss norms

[0055] The L1 norm loss function and the L2 norm loss function can be used in the robustness evaluation of autonomous navigation to calculate the difference between the actual value of autonomous navigation before and after adding noise and interference and without adding noise and interference.

[0056]

[0057] In the formula, y i and f(x i ) are the ordinates of the theoretical path and the actual path respectively: Same as the principle of using L1 norm loss function and L2 norm loss function in generalization evaluation, when performing robustness evaluation of autonomous navigation, if the difference between the target values ​​before and after adding noise is large, the L1 norm should be selected for calculation, otherwise the L2 norm should be selected for calculation.

[0058] 3.3 Pearson correlation coefficient

[0059] The Pearson correlation coefficient refers to the strength of the linear correlation between two variables. The larger the absolute value of r, the stronger the correlation. It can be used to judge the quality of trajectory overlap. The value range of r is [-1,1]. r>0 indicates positive correlation, r<0 indicates negative correlation, and r=0 indicates non-linear relationship.

[0060]

[0061] The overall correlation coefficient r is defined as the ratio of the covariance between two trajectories X and Y to the product of their standard deviations.

[0062] 3.4 Coefficient of determination COD

[0063] The coefficient of determination is a numerical characteristic that represents the relationship between one variable and another variable. It is a statistical indicator used to reflect the reliability of the changes in the dependent variables of two models. It is generally represented by the symbol "R" and can be defined as the ratio of the variation of the independent variable explained by all the independent variables in the model to the total variation of the independent variable.

[0064] First, there is a set of sample data sets (x1, y1), (x2, y2), ... (x n ,y n ), the predicted value is obtained by model calculation Observed data mean Then substitute it into the following formula to get RSS and TSS, and then get the coefficient of determination R.

[0065]

[0066] TSS is the total sum of squares, RSS is the residual sum of squares, and the larger the COD value, the closer the observed value is to the calculated value, which means that the anti-interference ability is stronger.

[0067] 4. Compare the two differences obtained in step 3 and obtain the evaluation score to evaluate the robustness of autonomous navigation.

[0068] The technical indicators for robustness evaluation of autonomous functions of intelligent navigation systems are mainly composed of interference loss IC, loss norm function, Pearson correlation coefficient, and coefficient of determination COD. The interference loss is used to characterize the loss of time, path and energy caused by the introduction of interference. The norm of the loss function characterizes the gap between the target value and the actual value before and after the interference is added. The Pearson correlation coefficient is used to measure the strength of the linear relationship between the target value and the actual value, and its value is between [-1,1]. The coefficient of determination COD is used to characterize the gap between observed data (actual data) and predicted data (theoretical data). The larger the COD value, the stronger the anti-interference ability. Figure 3 Shown is a schematic diagram of the fusion of autonomous navigation robustness indicators.

[0069] The above four indicators are weighted and integrated to obtain the robustness capability A of autonomous navigation. robust , as shown in formula (6). In the formula, r is the Pearson correlation coefficient, is the inverse of the interference loss, 1, R is the L1 / L2 norm loss function 2 is the coefficient of determination;

[0070]

[0071] For the robustness evaluation of autonomous navigation, we use a percentage system to evaluate the obtained A robust The evaluation principle of robustness evaluation is different from that of generalization evaluation, which only needs to be based on a pair of target value and actual value. The evaluation principle of robustness evaluation is to compare the target value before adding interference with the target value after adding interference and calculate the corresponding value A. robust1 , and then compare the actual value before adding the interference with the actual value after adding the interference, and calculate the corresponding value A robust2 , then A robust1 and A robust2 For comparison, the robustness scoring formula is shown in formula (7).

[0072]

[0073] It can be seen that if the score is closer to 100 points, that is, the closer the robustness of the actual system is to the robustness calculated by the target value, the better the system's autonomous navigation robustness is. If the score exceeds 100 points, it means that the interference of the scene is unreasonable and needs to be re-evaluated.

[0074] The autonomous navigation system robustness evaluation method proposed in this embodiment is suitable for comprehensive robustness evaluation in different scenarios and tasks. The selected indicators can comprehensively describe the navigation performance of the autonomous navigation system in time and space, and can quantitatively give the evaluation results.

[0075] The above is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technician familiar with this profession can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent replacement and improvement made to the above embodiments without departing from the content of the technical solution of the present invention, based on the technical essence of the present invention, within the spirit and principles of the present invention, still fall within the protection scope of the technical solution of the present invention.

Claims

1. A method for evaluating the robustness of an autonomous navigation system, characterized in that: The method comprises the following steps: Step 1: Global path planning in a specified original scene obtains the expected target value and the actual value of autonomous navigation; Step 2: By adding interference and noise, a scene after adding interference is generated, and on this basis, global path planning is performed to obtain the target value after adding interference and the actual value of autonomous navigation; Step 3, compare the target value before adding the interference with the target value after adding the interference, and calculate the corresponding value; then compare the actual value before adding the interference with the actual value after adding the interference, and calculate the corresponding value; Step 4: Compare the two differences obtained in step 3 to obtain an evaluation score.

2. The method for evaluating robustness of an autonomous navigation system according to claim 1, characterized in that: In step 3, the indicators based on the comparison of target values ​​before and after the interference and the comparison of actual values ​​before and after the interference are calculated by four methods: interference loss, L1 and L2 loss functions, Pearson correlation coefficient and determination coefficient.

3. The method for evaluating robustness of an autonomous navigation system according to claim 2, characterized in that: The interference loss includes the consumption of time, path and energy. The change in time consumption is the change in the time consumption after the interference is added and the time consumption before the interference is added. The change in path consumption is the change in the path consumption after the interference is added and the path consumption before the interference is added. The change in energy consumption refers to the change in the energy consumption after the interference is added and the energy consumption before the interference is added. The calculation is as follows: The change in consumption after the interference is added is divided by the value before the interference is added to get the change rate, and then the change rates are summed. Where ΔT is the change in time consumption after the interference is added; E T is the change in time consumption before interference is added; ΔS is the change in path consumption after interference is added; E S is the change in path consumption before interference is added; ΔE is the change in energy consumption after interference is added; E E is the change in energy consumption before interference is added.

4. The method for evaluating robustness of an autonomous navigation system according to claim 2, characterized in that: The L1 and L2 loss functions are calculated as follows, y i and f(x i ) are the ordinates of the theoretical path and the actual path respectively: The difference between the actual value of autonomous navigation before and after adding noise and interference and without adding noise and interference is calculated through the L1 norm loss function and the L2 norm loss function; The same principle as using the L1 norm loss function and the L2 norm loss function in generalization evaluation is used in the robustness evaluation of autonomous navigation. If the difference between the target values ​​before and after adding noise is large, the L1 norm should be selected for calculation, otherwise the L2 norm should be selected for calculation.

5. The method for evaluating robustness of an autonomous navigation system according to claim 2, characterized in that: The Pearson correlation coefficient, r, has a larger absolute value, indicating a stronger correlation, which can be used to judge the quality of trajectory overlap; the value range of r is [-1,1], r>0 indicates a positive correlation, r<0 indicates a negative correlation, and r=0 indicates a non-linear relationship; The overall correlation coefficient r is defined as the ratio of the covariance between two trajectories X and Y to the product of their standard deviations.

6. The method for evaluating robustness of an autonomous navigation system according to claim 2, characterized in that: The coefficient of determination is calculated as follows: First, there is a set of sample data sets (x1, y1), (x2, y2), ... (x n ,y n ), the predicted value is obtained by model calculation Observed data mean Then substitute it into the following formula to get RSS and TSS, and then get the coefficient of determination R; TSS is the total sum of squares, RSS is the residual sum of squares. The larger the value of the coefficient of determination, the closer the observed value is to the calculated value, which means that the anti-interference ability is stronger.

7. A method for evaluating robustness of an autonomous navigation system according to any one of claims 3 to 6, characterized in that: The above four indicators are weighted and integrated to obtain the robustness of autonomous navigation A. robust , as shown in formula (6), r is the Pearson correlation coefficient, is the inverse of the interference loss, 1, R is the L1 / L2 norm loss function 2 is the coefficient of determination; The A obtained by using the percentage system robust The value is scored, and the score is used to indicate the generalization quality; Compare the target value before adding the interference with the target value after adding the interference and calculate the corresponding value A robust1 , and then compare the actual value before adding the interference with the actual value after adding the interference, and calculate the corresponding value A robust2 , then A robust1 and A robust2 For comparison, the robustness scoring formula is shown in formula (7): If the score is closer to 100, that is, the closer the robustness of the actual system is to the robustness calculated by the target value, the better the system's autonomous navigation robustness is. If the score exceeds 100, it means that the interference of the scene is unreasonable and needs to be re-evaluated.

Images