A robustness evaluation method for autonomous navigation systems

Abstract

A method for evaluating the robustness of an autonomous navigation system. To achieve accurate, comprehensive, and quantitative evaluation of the robustness of an autonomous navigation system, and thereby provide a basis and direction for the research and improvement of autonomous navigation technology, the present invention summarizes the robustness evaluation indicators of an autonomous navigation system; establishes an indicator calculation method based on the comparison of target values ​​before and after interference and the comparison of actual values ​​before and after interference; and on this basis, fuses the indicators to obtain an evaluation score, thereby achieving the evaluation of the robustness of autonomous navigation. The present invention is suitable for comprehensive robustness evaluation in different scenarios and tasks. The selected indicators can comprehensively describe the navigation performance of the autonomous navigation system in time and space, and can quantitatively give the evaluation results. The present invention belongs to the research field of autonomous navigation system evaluation technology.

Description

Technical Field

[0001] The invention relates to a method for evaluating the robustness of an autonomous navigation system, and belongs to the research field of autonomous navigation system evaluation technology. Background Art

[0002] An ideal unmanned system should have the ability to sense the external environment with the help of sensors without external interference, and thus move autonomously to the target area. Autonomous navigation technology is the core technology to achieve this goal. Intelligent entities with autonomous navigation capabilities can be collectively referred to as autonomous navigation systems, such as autonomous mobile robots, autonomous unmanned submarines, etc.

[0003] Robustness refers to a system's ability to withstand failures and disturbances, specifically its ability to maintain stable operation or recover to a stable state in the face of various disturbances and uncertainties. Autonomous navigation systems often operate in complex and changing environments, performing a wide range of tasks. They must maintain the highest possible stability to complete their assigned tasks in the face of environmental disturbances and uncertainties, placing high demands on system robustness. Therefore, conducting an accurate and reasonable performance evaluation of the robustness of autonomous navigation systems is crucial.

[0004] However, current research on robustness evaluation of autonomous navigation systems has the following main problems:

[0005] (1) Such as quantitative description of external environment changes and interference to autonomous navigation systems;

[0006] (2) How to quantitatively calculate the evaluation indicators that reflect the robustness of autonomous navigation;

[0007] (3) How to integrate various evaluation indicators to form the final robustness evaluation results.

[0008] Existing performance evaluation methods mostly evaluate static targets and calculate various evaluation indicators by measuring their physical information. However, as a mobile target, the autonomous navigation system will generate a large amount of data in time and space. To comprehensively evaluate its robustness, it is necessary to rationally utilize the path and time data generated during the autonomous navigation process, which puts forward new requirements for the calculation of evaluation indicators and the construction of the evaluation system. Summary of the Invention

[0009] In order to solve the above problems existing in the prior art, the present invention further proposes a robustness evaluation method for an autonomous navigation system.

[0010] The technical solution adopted by the present invention to solve the above problems is:

[0011] The present invention comprises the steps of:

[0012] Step 1: Global path planning in the specified original scene obtains the expected target value and the actual value of autonomous navigation;

[0013] Step 2: By adding interference and noise, a scene after interference is generated. On this basis, global path planning is performed to obtain the target value after interference addition and the actual value of autonomous navigation;

[0014] Step 3: Compare the target value before adding the interference with the target value after adding the interference, and calculate the corresponding value; then compare the actual value before adding the interference with the actual value after adding the interference, and calculate the corresponding value;

[0015] Step 4: Compare the two differences obtained in step 3 to obtain an evaluation score.

[0016] Furthermore, the indicators based on the comparison of target values ​​before and after interference and the comparison of actual values ​​before and after interference are calculated through interference loss, L1 and L2 loss functions, Pearson correlation coefficient and coefficient of determination COD.

[0017] Furthermore, the interference loss includes time, path, and energy consumption. The time consumption change is the change in time consumption after the interference is added compared to the time consumption before the interference is added. The path consumption change is the change in path consumption after the interference is added compared to the path consumption before the interference is added. The energy consumption change refers to the change in energy consumption after the interference is added compared to the energy consumption before the interference is added. The calculation is as follows:

[0018] The change in consumption after the interference is added is divided by the value before the interference is added to get the change rate, and then the change rates are summed.

[0019]

[0020] Where ΔT is the change in time consumption after the interference is added; E T is the change in time consumption before interference is added; ΔS is the change in path consumption after interference is added; E S is the change in path consumption before interference is added; ΔE is the change in energy consumption after interference is added; E E is the change in energy consumption before interference is added.

[0021] Furthermore, the L1 and L2 loss functions are calculated as follows: i and f(x i ) are the ordinates of the theoretical path and the actual path respectively: the difference between the actual values ​​of autonomous navigation before and after adding noise and interference and without adding noise and interference is calculated by L1 norm loss function and L2 norm loss function;

[0022]

[0023] Similar to the principle of using the L1 norm loss function and the L2 norm loss function in generalization evaluation, when performing robustness evaluation of autonomous navigation, if the difference between the target values ​​before and after adding noise is large, the L1 norm should be selected for calculation, otherwise the L2 norm should be selected for calculation.

[0024] Furthermore, the Pearson correlation coefficient r, the larger the absolute value, the stronger the correlation, which can be used to judge the quality of trajectory overlap; the value range of r is [-1, 1], r>0 indicates positive correlation, r<0 indicates negative correlation, and r=0 indicates non-linear relationship;

[0025]

[0026] The overall correlation coefficient r is defined as the ratio of the covariance between two trajectories X and Y to the product of their standard deviations.

[0027] Furthermore, the coefficient of determination is calculated as follows:

[0028] First, there is a set of sample data sets (x1, y1), (x2, y2), ... (x n ,y n ), the predicted value is obtained by model calculation Observation data mean Then substitute into the following formula to get RSS and TSS, and then get the coefficient of determination R;

[0029]

[0030] TSS is the total sum of squares, RSS is the residual sum of squares. The larger the coefficient of determination value is, the closer the observed value is to the calculated value, which means the stronger the anti-interference ability is.

[0031] Furthermore, the above four indicators are weighted and integrated to obtain the robustness of autonomous navigation A. robust , as shown in formula (6), r is the Pearson correlation coefficient, is the inverse of the interference loss, 1, R is the L1 / L2 norm loss function 2 is the coefficient of determination;

[0032]

[0033] A is calculated using the percentage system robust The value is scored, and the score is used to indicate the generalization quality;

[0034] Compare the target value before adding the interference with the target value after adding the interference and calculate the corresponding value Arobust1 , then compare the actual value before adding the interference with the actual value after adding the interference, and calculate the corresponding value A robust2 , then A robust1 and A robust2 For comparison, the robustness scoring formula is shown in formula (7):

[0035]

[0036] The closer the score is to 100, that is, the closer the robustness of the actual system is to the robustness calculated from the target value, the better the robustness of the system's autonomous navigation. If the score exceeds 100, it means that the interference in the scene is unreasonable and needs to be re-evaluated.

[0037] The beneficial effects of the present invention are:

[0038] 1. The present invention summarizes the robustness evaluation indicators of the autonomous navigation system, establishes an indicator calculation method based on the comparison of target values ​​before and after interference and the comparison of actual values ​​before and after interference, and on this basis, performs indicator fusion to obtain the robustness evaluation result of the autonomous navigation system.

[0039] 2. The autonomous navigation system robustness evaluation method proposed in this invention is suitable for comprehensive robustness evaluation in different scenarios and tasks. The selected indicators can comprehensively describe the navigation performance of the autonomous navigation system in time and space, and can quantitatively give the evaluation results.

[0040] 3. The present invention realizes an accurate and comprehensive evaluation of the robustness of the autonomous navigation system, thereby providing a basis and direction for the research and improvement of autonomous navigation technology. At the same time, the present invention also has certain reference significance for performance evaluation research in other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is the overall flow chart of the present invention;

[0042] Figure 2 Schematic diagram of the robustness evaluation index design scheme of the present invention;

[0043] Figure 3 It is a schematic diagram of the fusion of robustness indicators of the autonomous navigation system of the present invention. DETAILED DESCRIPTION

[0044] The autonomous navigation system robustness evaluation method described in this embodiment includes the following steps:

[0045] Step 1: Global path planning in the specified original scene obtains the expected target value and the actual value of autonomous navigation;

[0046] The target value of an autonomous navigation system's capability assessment is the expected value obtained by a global path planning algorithm under a specified scenario, including data such as the path point set, path length, path energy consumption, and path time consumption. For unmanned vehicles, for example, the A* algorithm, due to its ability to obtain the optimal path, is used for path planning and obtains the target path point set. Based on this target path point set, target values ​​such as path length, path energy consumption, and path time consumption are calculated.

[0047] Step 2: By adding interference and noise, a scene after interference is generated, and then global path planning is performed on this basis to obtain the target value after interference addition and the actual value of autonomous navigation;

[0048] Step 3: Compare the target value before adding the interference with the target value after adding the interference, and calculate the corresponding value; then compare the actual value before adding the interference with the actual value after adding the interference, and calculate the corresponding value;

[0049] According to the data characteristics of the autonomous navigation system path point set, path length, path energy consumption, path time consumption, etc., the robustness evaluation index can be calculated. On this basis, the calculation results of the target value index and the actual value index calculation results are compared to obtain the robustness evaluation results. Therefore, appropriate technical indicators should be selected. Figure 2 As shown in the figure, the technical indicator design scheme of the robustness evaluation method is formulated. According to the relationship between the target value and the target value, and the actual value and the actual value after the interference is added, four different evaluation indicator calculation rules are formulated, namely interference loss, L1 and L2 loss function, Pearson correlation coefficient, and coefficient of determination COD. They will be introduced below respectively.

[0050] 3.1 Interference Loss

[0051] Interference consumption is defined as the time, path, and energy losses caused by the introduction of interference. Time consumption change refers to the difference between the time consumption before and after the interference is introduced. Path consumption change refers to the difference between the path consumption after and before the interference is introduced. Energy consumption change refers to the difference between the energy consumption before and after the interference is introduced.

[0052] Considering that the change in path length, time, and energy consumption are of different dimensions, the change needs to be dimensionless, that is, the change in consumption after the interference is added is divided by the value before the interference is added to obtain the change rate, and then the change rates are summed.

[0053]

[0054] 3.2 L1 and L2 loss norms

[0055] The L1-norm loss function and the L2-norm loss function can be used in the robustness evaluation of autonomous navigation to calculate the difference between the actual value of autonomous navigation before and after adding noise and interference and without adding noise and interference.

[0056]

[0057] Where y i and f(x i ) are the ordinates of the theoretical path and the actual path, respectively: Same as the principle of using L1 norm loss function and L2 norm loss function in generalization evaluation, when performing robustness evaluation of autonomous navigation, if the difference between the target values ​​before and after adding noise is large, the L1 norm should be selected for calculation, otherwise the L2 norm should be selected for calculation.

[0058] 3.3 Pearson correlation coefficient

[0059] The Pearson correlation coefficient measures the strength of the linear correlation between two variables. A larger absolute value of r indicates a stronger correlation and can be used to determine the quality of trajectory overlap. r ranges from -1 to 1, with r > 0 indicating a positive correlation, r < 0 indicating a negative correlation, and r = 0 indicating a non-linear relationship.

[0060]

[0061] The overall correlation coefficient r is defined as the ratio of the covariance between two trajectories X and Y to the product of their standard deviations.

[0062] 3.4 Coefficient of determination COD

[0063] The coefficient of determination is a numerical characteristic that represents the relationship between one variable and another. It is a statistical indicator used to reflect the reliability of the changes in the dependent variables of two models. It is generally represented by the symbol "R" and can be defined as the ratio of the variation of the independent variable explained by all the independent variables in the model to the total variation of the independent variables.

[0064] First, there is a set of sample data sets (x1, y1), (x2, y2), ... (x n ,y n ), the predicted value is obtained by model calculation Observation data mean Then substitute into the following formula to obtain RSS and TSS, and then obtain the coefficient of determination R.

[0065]

[0066] TSS is the total sum of squares, RSS is the residual sum of squares, and the larger the COD value, the closer the observed value and the calculated value are, which means the stronger the anti-interference ability.

[0067] 4. Compare the two differences obtained in step 3 to obtain an evaluation score to evaluate the robustness of autonomous navigation.

[0068] The technical indicators for robustness evaluation of autonomous functions of intelligent navigation systems are mainly composed of interference loss IC, loss norm function, Pearson correlation coefficient, and coefficient of determination COD. Among them, interference loss is used to characterize the loss of time, path and energy caused by the introduction of interference. The norm of the loss function characterizes the size of the gap between the target value and the actual value before and after the interference is added. The Pearson correlation coefficient is used to measure the strength of the linear relationship between the target value and the actual value, and its value is between [-1,1]. The coefficient of determination COD is used to characterize the gap between observed data (actual data) and predicted data (theoretical data). The larger the COD value, the stronger the anti-interference ability. Figure 3 Shown is a schematic diagram of the fusion of autonomous navigation robustness indicators.

[0069] The above four indicators are weighted and integrated to obtain the robustness of autonomous navigation A. robust , as shown in formula (6). In the formula, r is the Pearson correlation coefficient, is the inverse of the interference loss, 1, R is the L1 / L2 norm loss function 2 is the coefficient of determination;

[0070]

[0071] For the robustness evaluation of autonomous navigation, we use the percentage system to evaluate the obtained A robust The evaluation principle of robustness evaluation is different from that of generalization evaluation, which is based on a pair of target value and actual value. The evaluation principle of robustness evaluation is to compare the target value before adding interference with the target value after adding interference and calculate the corresponding value A. robust1 , then compare the actual value before adding the interference with the actual value after adding the interference, and calculate the corresponding value A robust2 , then A robust1 and A robust2 For comparison, the robustness scoring formula is shown in formula (7).

[0072]

[0073] It can be seen that the closer the score is to 100, that is, the closer the robustness of the actual system is to the robustness calculated from the target value, the better the robustness of the system's autonomous navigation. If the score exceeds 100, it means that the interference of the scene is unreasonable and needs to be re-evaluated.

[0074] The autonomous navigation system robustness evaluation method proposed in this embodiment is suitable for comprehensive robustness evaluation in different scenarios and tasks. The selected indicators can comprehensively describe the navigation performance of the autonomous navigation system in time and space, and can quantitatively give the evaluation results.

[0075] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the present invention has been disclosed as a preferred embodiment as above, it is not intended to limit the present invention. Any technician familiar with the present profession can make some changes or modifications to equivalent embodiments of equivalent changes using the technical content disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent replacement and improvement of the above embodiments made according to the technical essence of the present invention, within the spirit and principles of the present invention, without departing from the content of the technical solution of the present invention, shall still fall within the scope of protection of the technical solution of the present invention.

Claims

1. A robustness evaluation method for an autonomous navigation system, characterized in that: The method comprises the following steps: Step 1: Global path planning in the specified original scene obtains the expected target value and the actual value of autonomous navigation; Step 2: By adding interference and noise, a scene after interference is generated. On this basis, global path planning is performed to obtain the target value after interference addition and the actual value of autonomous navigation; Step 3: Compare the target value before adding the interference with the target value after adding the interference, and calculate the corresponding value; then compare the actual value before adding the interference with the actual value after adding the interference, and calculate the corresponding value; Step 4: Compare the two differences obtained in step 3 to obtain an evaluation score.

2. The method for robustness evaluation of an autonomous navigation system according to claim 1, characterized in that: In step 3, the indicators based on the comparison of target values ​​before and after the interference and the comparison of actual values ​​before and after the interference are calculated using four methods: interference loss, L1 and L2 loss functions, Pearson correlation coefficient, and determination coefficient.

3. The method for robustness evaluation of an autonomous navigation system according to claim 2, characterized in that: The interference loss includes time, path, and energy consumption. The time consumption change is the change in time consumption after the interference is added compared to the time consumption before the interference is added. The path consumption change is the change in path consumption after the interference is added compared to the path consumption before the interference is added. The energy consumption change refers to the change in energy consumption after the interference is added compared to the energy consumption before the interference is added. The calculation is as follows: The change rate is obtained by dividing the change in consumption after the interference by the value before the interference, and then summing the change rates. (1) Where, is the change in time consumption after interference is added; is the change in time consumption before interference is added; is the change in path consumption after interference is added; is the change in path consumption before interference is added; is the change in energy consumption after interference is added; is the change in energy consumption before interference is added.

4. The method for robustness evaluation of an autonomous navigation system according to claim 2, wherein: The L1 and L2 loss functions are calculated as follows, and They are the ordinates of the theoretical path and the actual path respectively: The difference between the actual value of autonomous navigation before and after adding noise and interference and without adding noise and interference is calculated through the L1 norm loss function and the L2 norm loss function; (2) (3) Similar to the principle of using the L1 norm loss function and the L2 norm loss function in generalization evaluation, when performing robustness evaluation of autonomous navigation, if the difference between the target values ​​before and after adding noise is large, the L1 norm should be selected for calculation, otherwise the L2 norm should be selected for calculation.

5. The method for robustness evaluation of an autonomous navigation system according to claim 2, wherein: The Pearson correlation coefficient, r, has a larger absolute value, indicating a stronger correlation, which can be used to judge the quality of trajectory overlap; the value range of r is [-1, 1], r>0 indicates a positive correlation, r<0 indicates a negative correlation, and r=0 indicates a non-linear relationship; (4) The overall correlation coefficient r is defined as the ratio of the covariance between two trajectories X and Y to the product of their standard deviations.

6. The method for robustness evaluation of an autonomous navigation system according to claim 2, characterized in that: The coefficient of determination is calculated as follows: First, there is a set of sample data sets , the predicted value is obtained by model calculation , the mean of the observed data ; Then substitute into the following formula to get RSS and TSS , and then get the coefficient of determination R； (5) TSS is the total sum of squares, RSS is the residual sum of squares. The larger the coefficient of determination value is, the closer the observed value is to the calculated value, which means the stronger the anti-interference ability is.

7. The method for robustness evaluation of an autonomous navigation system according to any one of claims 3 to 6, characterized in that: The above four indicators are weighted and integrated to obtain the robustness of autonomous navigation. A robust , as shown in formula (6), is the Pearson correlation coefficient, is the inverse of the interference loss, is the inverse of the norm loss function, is the coefficient of determination; (6) The percentage system is used to obtain A robust The value is scored, and the score is used to indicate the generalization quality; Compare the target value before adding the interference with the target value after adding the interference and calculate the corresponding value A robust1 , then compare the actual value before adding the interference with the actual value after adding the interference, and calculate the corresponding value A robust2 , then A robust1 and A robust2 For comparison, the robustness scoring formula is shown in formula (7): (7) The closer the score is to 100, that is, the closer the robustness of the actual system is to the robustness calculated from the target value, the better the robustness of the system's autonomous navigation. If the score exceeds 100, it means that the interference in the scene is unreasonable and needs to be re-evaluated.

Images