Robot evaluation method based on minimum scene

Abstract

The invention relates to the technical field of robot evaluation, and discloses a robot evaluation method based on a minimum scene, which comprises the following steps: defining the environmental parameters of the minimum scene, and completing the construction of the minimum scene; setting an evaluation index in the minimum scene, testing the robot on the basis of the minimum scene, obtaining related test data required by the evaluation index corresponding to the minimum scene, and obtaining the weight of the evaluation index in the minimum scene by using a parameter optimization model; coupling the minimum scene to obtain a complex scene, setting an evaluation index under the complex scene, obtaining test data under the complex scene, and obtaining a weight corresponding to the minimum scene in the complex scene through a weight information entropy fusion mode. According to the invention, by designing and evaluating a plurality of minimum scenes, the performance of the robot under different functions and environments is comprehensively investigated. Meanwhile, a plurality of minimum scenes are coupled to form a complex scene, so that an actual application scene can be better simulated, and more accurate performance evaluation is provided.

Description

Technical Field

[0001] The present invention relates to the technical field of robot evaluation, and in particular to a robot evaluation method based on a minimum scenario. Background Art

[0002] With the development of robots, their functions and performance have become more and more perfect, and the demand for accurate evaluation of robot performance and reliability has gradually increased. In the process of robot development and design, it is crucial to evaluate the performance and reliability of the robot system in the application environment. However, there are obvious defects in directly evaluating the performance and reliability of robots in complex scenarios. Whether in simulation or in real environment, directly evaluating the performance and reliability of robots in complex scenarios can reveal the capabilities and limitations of robot systems. However, when evaluating directly in complex scenarios, on the one hand, the economic and time costs are high. On the other hand, in simple or minimal scenarios, the test tasks are single and simple. Under the same test task, the test results are highly consistent, and it is easy to process and annotate the data according to the judgment rules. However, there are too many uncontrollable factors in complex scenarios. When robots perform the same test tasks in complex scenarios, the consistency and similarity of task execution cannot be guaranteed, and there is a lot of uncertainty in the test results. This makes it impossible to process and annotate the test data based on certain judgment rules or standard levels, and it is difficult to evaluate the performance of robots.

[0003] The present invention mainly aims at the shortcomings of the existing technology in cost-effectiveness, safety assessment efficiency and accuracy, and uncertainty of test results for the same test task when directly evaluating the performance and reliability of the robot in complex scenarios. A method for completing the complex scenario evaluation of the robot based on the minimum scenario evaluation is proposed in the present invention. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a robot evaluation method based on a minimum scenario.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0006] A robot evaluation method based on a minimal scenario, comprising:

[0007] Define the minimum scene environment parameters based on the robot evaluation task and test standards to complete the construction of the minimum scene; the minimum scene can meet the requirements for performance testing of a certain function of the robot;

[0008] Set the evaluation index x under the minimum scenario ij , the evaluation index x in the minimum scenario ijIt is used to measure the test performance of the jth function of the robot in the i-th minimum scenario, 1≤j≤J, J represents the total number of functions of the robot; the robot is tested on the basis of the minimum scenario, the test data required for the evaluation index corresponding to the minimum scenario is obtained, the test data is annotated, and the evaluation index x under the minimum scenario is obtained by using the parameter optimization model ij The weight w ij , complete the evaluation of the robot in the minimum scenario;

[0009] Couple I minimum scenarios to obtain a complex scenario, and set the evaluation index X under the complex scenario j , obtain the test data in the complex scene and transfer the evaluation index in the minimum scene to the evaluation index in the complex scene by weight information entropy fusion, and obtain the weight W corresponding to the i-th minimum scene in the complex scene i , 1≤i≤I, complete the evaluation of the robot in complex scenarios.

[0010] Furthermore, the test data required for the evaluation index corresponding to the minimum scenario is obtained, the test data is labeled, and the evaluation index x under the minimum scenario is obtained by using the parameter optimization model. ij The weight w ij , including:

[0011] According to the set judgment rules, the obtained test data and related calculation results that are directly and / or indirectly related to the evaluation indicators are marked, and then the weights w of each evaluation indicator in the minimum scenario are obtained through the parameter optimization model. ij .

[0012] Furthermore, the I minimum scenarios are coupled to obtain a complex scenario, and the evaluation index X is set under the complex scenario. j , including:

[0013] Based on the robot evaluation task, the environmental parameters of the complex scene are defined on the basis of the minimum scene, and the complex scene is determined; the environmental parameters of the complex scene include all or part of the environmental parameters of the minimum scene; the evaluation index X under the complex scene j , including evaluation metrics for minimal scenarios;

[0014] The weight W of each minimum scene in the complex scene is determined by the weight information entropy fusion method. i ; Complete the evaluation of robots in complex scenarios.

[0015] Furthermore, the test data under the complex scene is obtained and the evaluation index under the minimum scene is transferred to the evaluation index under the complex scene by means of weight information entropy fusion, so as to obtain the weight W corresponding to the i-th minimum scene in the complex scene. i, including:

[0016] Based on the principle of information entropy invariance, the equation is established:

[0017] ∑w ij *h ij *W i =H j ;

[0018] Among them, h ij Represents the evaluation index x based on the minimum scenario ij The calculated information entropy is used to measure the degree of uncertainty of data:

[0019] h ij =-∑(x ij )*log 2(x ij );

[0020] H j Represents the evaluation index X based on complex scenarios j The calculated information entropy;

[0021] Solve the equation to obtain the weight W of each minimum scene in the complex scene i .

[0022] Furthermore, the minimum scenario can meet the requirements for performance testing of a certain function of the robot, specifically including:

[0023] One minimum scenario can meet the requirements for performance testing of a certain function of the robot, or multiple minimum scenarios can meet the requirements for performance testing of a certain function of the robot.

[0024] Compared with the prior art, the beneficial technical effects of the present invention are:

[0025] The present invention completes the robot evaluation through the minimum scenario, effectively improving the cost-effectiveness of robot evaluation and replacement. At the same time, in the minimum scenario, the influencing factors can be effectively controlled to improve the evaluation efficiency and accuracy. At the same time, based on the test in the minimum scenario, the consistency of the test results can be guaranteed, and the test data can be labeled according to certain judgment rules.

[0026] The present invention realizes the fusion of the robot's minimum scene and complex scene through the method of weighted information entropy fusion, obtains the evaluation result in the complex scene based on the minimum scene evaluation result, and solves the problem that the complex scene test data cannot be labeled according to certain judgment rules.

[0027] The present invention designs and evaluates multiple minimum scenarios to comprehensively examine the performance of the robot in different functions and environments. At the same time, by coupling multiple minimum scenarios to form a complex scenario, it can better simulate the actual application scenario and provide a more accurate performance evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is an overall schematic diagram of the robot evaluation method of the present invention;

[0029] Figure 2 is a flow chart of the robot evaluation method of the present invention;

[0030] Figure 3a and Figure 3b They respectively represent a schematic diagram of the minimum scenario test evaluation process and a schematic diagram of the complex scenario test evaluation process of the present invention. DETAILED DESCRIPTION

[0031] A preferred embodiment of the present invention is described in detail below with reference to the accompanying drawings.

[0032] This embodiment takes the evaluation of the autonomous navigation performance of a ground mobile robot as an example, and the robot evaluation method specifically includes the following steps:

[0033] S1, define the minimum scene environment parameters and determine the minimum scene:

[0034] Based on the autonomous navigation performance evaluation task of the ground mobile robot, the minimum scene required by the evaluation task is defined, and the environmental parameters of the minimum scene are determined. The environmental parameters of the minimum scene include physical environment parameters, weather environment parameters, interference environment parameters, etc., among which the physical environment parameters include: terrain parameters, space parameters, obstacle parameters, etc.; weather environment parameters include: lighting parameters, humidity parameters, temperature parameters, etc.; interference environment parameters include: strong light parameters, smoke parameters, dust parameters, etc., and its minimum scene can cover the test requirements of the autonomous navigation performance of the ground mobile robot.

[0035] After clarifying the minimum scene environment parameters, the minimum scene is determined based on the ground mobile robot autonomous navigation performance evaluation task. The minimum scene includes but is not limited to:

[0036] Component-level minimum test scenarios, i.e., minimum scenarios for devices and algorithms for autonomous navigation of ground mobile robots, such as: minimum scenarios for lidar performance testing, minimum scenarios for perception algorithms, etc.

[0037] Minimum obstacle avoidance scenario, minimum straight path scenario, minimum constant curvature path scenario, minimum variable curvature path scenario, minimum undulating terrain scenario, etc. Minimum interference-free scenario; the environmental complexity of the scenario is consistent or similar;

[0038] The scenes with minimal interference include scenes with minimal strong light interference, scenes with minimal weak light interference, etc.

[0039] S2, clarify the minimum scenario evaluation indicators:

[0040] After determining the minimum scenario for the autonomous navigation performance test of the ground mobile robot, the evaluation index x of each minimum scenario is clarified. ij , x ij Used to measure the test performance of the jth function of the robot in the i-th minimum scenario.

[0041] In a preferred embodiment, the minimum scenario-related evaluation indicators (not limited to the following minimum scenarios and evaluation indicators) are as follows:

[0042] The minimum scene evaluation indicators for component-level lidar performance testing include: lidar ranging capability, distance accuracy, distance precision, angle resolution, angle accuracy, angle accuracy and other evaluation indicators; the minimum scene evaluation indicators for perception algorithms include: absolute trajectory error (ATE), relative pose error (RPE), etc.

[0043] The evaluation indicators of the minimum obstacle avoidance scenario include: the obstacle avoidance reaction distance of the ground mobile robot, the distance between the obstacle and the robot (when parallel to the obstacle), the distance to escape from the obstacle, the narrowest distance between two obstacles that can be passed, mileage, etc.; the minimum scenarios of straight lines, constant curvature, and variable curvature paths: ATE, RPE, completion time T, path smoothness, mileage, etc.; the minimum scenario of undulating terrain: the maximum slope that can be passed, the stability of the robot's posture, etc.

[0044] The evaluation indicators of the scenes with minimal interference, such as the scene with minimal strong light interference and the scene with minimal weak light interference, include: map completion, positioning accuracy, etc.

[0045] S3, minimum scenario test, obtain the weight corresponding to the minimum scenario evaluation index:

[0046] Complete the establishment of the minimum scene in a simulation environment and / or a real environment, and obtain simulation test data / real test data that is directly and / or related to the minimum scene evaluation indicators obtained in step S2 through the test equipment; such as obtaining the minimum scene, mileage, and posture, speed, acceleration, etc. of a ground mobile robot completing a certain minimum scene.

[0047] The acquired data is processed, that is, the test data indirectly related to the evaluation indicators are calculated, and then the processed data is labeled according to the minimum scene evaluation criteria, and the weights w of each evaluation indicator in the minimum scene are determined through the trained parameter optimization model. ij , complete the evaluation of the minimum scenario.ij Indicates the evaluation index x in the minimum scenario ij The corresponding weight.

[0048] S4, minimum scene coupling, determine complex scenes:

[0049] Repeat the above steps to complete the evaluation of other minimum scenes. After completing the evaluation of each minimum scene, modify each minimum scene appropriately according to the evaluation results, such as the degree of strong light interference, maximum slope and other environmental parameters. Then, couple the minimum scenes to obtain a complex scene. The complex scene includes one or more complete or incomplete minimum environments determined in the first step, and can be randomly arranged in the complex scene, so that the coupled complex scene is more in line with the actual working conditions of the autonomous navigation of the ground mobile robot.

[0050] S5, clarify the evaluation indicators for complex scenarios:

[0051] After the complex scene is determined, the evaluation index X of the ground mobile robot in the complex environment is determined. j (X j represents the jth evaluation index in a complex scenario), such as completion time T, absolute trajectory error, relative pose error, map building completeness, mileage, obstacle avoidance reaction distance, etc. According to step S3 and step S4, simulation test data / real test data directly and / or indirectly related to the evaluation index in a complex environment is obtained, and the obtained data is processed, that is, the test data indirectly related to the evaluation index is calculated to obtain the data of the corresponding evaluation index.

[0052] S6, complex scenario test, determine the minimum weight of each scenario:

[0053] According to the weight information fusion algorithm, the weight W of each minimum scene in the complex scene is determined i , W i Indicates the weight corresponding to the i-th minimum scene in building a complex scene;

[0054] Specifically:

[0055] According to the information entropy calculation formula h ij =-∑(x ij )*log 2(x ij ) Determine the minimum scene evaluation index x contained in the complex scene ij The corresponding information entropy h ij Similarly, the evaluation index X of the complex environment is calculated j The corresponding information entropy H j , the minimum scenario evaluation index x ij It has been determined in the minimum scenario evaluation; based on the principle of information entropy invariance, the equation ∑w is established ij*h ij *W i =H j , where H j With h ij are the evaluation indicators X for the same robot function in complex scenarios and minimal scenarios respectively. j and x ij The corresponding information entropy; w ij is the evaluation index x in the minimum scenario ij According to the information entropy of each evaluation index determined by the complex scene and the minimum scene, the proportion of each minimum scene in the complex scene is obtained. i , and based on the minimum scene evaluation judgment rules, the test of the autonomous navigation performance of the ground mobile robot in complex scenes is obtained.

[0056] S7, complete the complex scenario test evaluation based on the minimum scenario.

[0057] In summary, the present invention first defines the minimum scene environment parameters, and the minimum scene can meet the requirements of a certain key function of the robot; secondly, the evaluation indicators under the minimum scene are set, and the evaluation indicators under the minimum scene measure a certain performance of the robot; then, the robot is tested on the basis of the minimum scene, and the relevant test data required for the indicators corresponding to the minimum scene are obtained, the test data is labeled, and the weights of each evaluation indicator in the minimum scene are obtained by using the parameter optimization model to complete the evaluation of the robot in the minimum scene; finally, on the basis of completing multiple minimum scenes, multiple minimum scenes are coupled to obtain a complex scene and the evaluation indicators under the complex scene are set, and the evaluation indicators under the minimum scene are migrated to the evaluation indicators under the complex scene by means of weight information entropy fusion, and the weights of each minimum scene in the complex scene are obtained, thereby completing the test and evaluation of the robot in the complex scene.

[0058] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention, and any reference numerals in the claims should not be regarded as limiting the claims involved.

[0059] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A robot evaluation method based on a minimum scenario, characterized in that: include: Define the minimum scene environment parameters based on the robot evaluation tasks and test standards, and complete the construction of the minimum scene; The minimum scenario can meet the requirements for performance testing of a certain function of the robot; Set the evaluation index x under the minimum scenario ij , the evaluation index x in the minimum scenario ij It is used to measure the test performance of the jth function of the robot in the i-th minimum scenario, 1≤j≤J, J represents the total number of functions of the robot; the robot is tested on the basis of the minimum scenario, the test data required for the evaluation index corresponding to the minimum scenario is obtained, the test data is annotated, and the evaluation index x under the minimum scenario is obtained by using the parameter optimization model ij The weight w ij , complete the evaluation of the robot in the minimum scenario; Couple I minimum scenarios to obtain a complex scenario, and set the evaluation index X under the complex scenario j , obtain the test data in the complex scene and transfer the evaluation index in the minimum scene to the evaluation index in the complex scene by weight information entropy fusion, and obtain the weight W corresponding to the i-th minimum scene in the complex scene i , 1≤i≤I, complete the evaluation of the robot in complex scenarios.

2. The robot evaluation method based on the minimum scenario according to claim 1, characterized in that: The method obtains the test data required for the evaluation index corresponding to the minimum scenario, labels the test data, and uses the parameter optimization model to obtain the evaluation index x under the minimum scenario. ij The weight w ij , specifically including: According to the set judgment rules, the obtained test data and related calculation results that are directly and / or indirectly related to the evaluation indicators are marked, and then the weights w of each evaluation indicator in the minimum scenario are obtained through the parameter optimization model. ij .

3. The robot evaluation method based on the minimum scenario according to claim 1, characterized in that: The I minimum scenarios are coupled to obtain a complex scenario, and the evaluation index X is set under the complex scenario. j , specifically including: Based on the robot evaluation task, the environmental parameters of the complex scene are defined on the basis of the minimum scene, and the complex scene is determined; the environmental parameters of the complex scene include all or part of the environmental parameters of the minimum scene; the evaluation index X under the complex scene j , including evaluation metrics for minimal scenarios; The weight W of each minimum scene in the complex scene is determined by the weight information entropy fusion method. i ; Complete the evaluation of robots in complex scenarios.

4. The robot evaluation method based on the minimum scenario according to claim 1, characterized in that: The test data under the complex scene is obtained and the evaluation index under the minimum scene is transferred to the evaluation index under the complex scene by means of weight information entropy fusion, and the weight W corresponding to the i-th minimum scene in the complex scene is obtained. i , specifically including: Based on the principle of information entropy invariance, the equation is established: ∑w ij *h ij *W i =H j ; Among them, h ij Represents the evaluation index x based on the minimum scenario ij The calculated information entropy is used to measure the degree of uncertainty of data: h ij =-∑(x ij )*log2(x ij ); H j Represents the evaluation index X based on complex scenarios j The calculated information entropy; Solve the equation to obtain the weight W of each minimum scene in the complex scene i .

5. The robot evaluation method based on the minimum scenario according to claim 1, characterized in that: The minimum scenario can meet the requirements for performance testing of a certain function of the robot, including: One minimum scenario can meet the requirements for performance testing of a certain function of the robot, or multiple minimum scenarios can meet the requirements for performance testing of a certain function of the robot.

Images