A robot evaluation method based on minimal scenarios

Abstract

The present invention relates to the field of robot evaluation technology, and discloses a robot evaluation method based on a minimum scene, comprising: defining minimum scene environmental parameters and completing the construction of the minimum scene; setting evaluation indicators under the minimum scene, testing the robot based on the minimum scene, obtaining relevant test data required for the evaluation indicators corresponding to the minimum scene, and using a parameter optimization model to obtain the weight of the evaluation indicator under the minimum scene; coupling the minimum scene to obtain a complex scene, setting evaluation indicators under the complex scene, obtaining test data under the complex scene, and obtaining the weight corresponding to the minimum scene in the complex scene through weight information entropy fusion. The present invention comprehensively examines the performance of the robot under different functions and environments by designing and evaluating multiple minimum scenes. At the same time, by coupling multiple minimum scenes to form a complex scene, it can better simulate actual application scenarios and provide more accurate performance evaluation.

Description

Technical Field

[0001] The present invention relates to the technical field of robot evaluation, and in particular to a robot evaluation method based on a minimum scenario. Background Art

[0002] As robots develop, their functionality and performance become increasingly sophisticated, leading to an increasing need for accurate performance and reliability assessments. During the robot development and design process, evaluating the performance and reliability of robotic systems within their application environments is crucial. However, directly evaluating robot performance and reliability in complex scenarios has significant drawbacks. While directly evaluating robot performance and reliability in complex scenarios, both in simulation and in real-world environments, can reveal the capabilities and limitations of robotic systems, conducting such evaluations directly in complex scenarios is both economically and time-consuming. Furthermore, simple or minimal scenarios offer single, straightforward test tasks, resulting in highly consistent test results and easy data processing and annotation based on judgment rules. However, complex scenarios present numerous uncontrollable factors, making it difficult to guarantee consistent and similar performance when robots perform the same test task in complex scenarios. This leads to significant uncertainty in test results, making it impossible to process and annotate test data based on specific judgment rules or standard levels, making it difficult to evaluate robot performance.

[0003] The present invention mainly addresses the shortcomings of existing technologies in directly evaluating robot performance and reliability in complex scenarios, such as cost-effectiveness, safety assessment efficiency and accuracy, and uncertainty in test results for the same test tasks. A method is proposed to complete the robot's complex scenario evaluation based on the minimum scenario evaluation. Summary of the Invention

[0004] In order to solve the above technical problems, the present invention provides a robot evaluation method based on a minimum scenario.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0006] A robot evaluation method based on a minimal scenario, including:

[0007] Define the minimum scenario environment parameters based on the robot evaluation task and test standards, and complete the construction of the minimum scenario; the minimum scenario can meet the requirements for performance testing of a certain function of the robot;

[0008] Set the evaluation index x under the minimum scenario ij , the evaluation index x in the minimum scenario ijIt is used to measure the test performance of the jth function of the robot in the i-th minimum scenario, 1≤j≤J, J represents the total number of functions of the robot; the robot is tested on the basis of the minimum scenario, the relevant test data required for the evaluation index corresponding to the minimum scenario is obtained, the test data is annotated, and the evaluation index x under the minimum scenario is obtained using the parameter optimization model ij The weight w ij , complete the evaluation of the robot in the minimum scenario;

[0009] Couple I minimum scenarios to obtain a complex scenario, and set the evaluation index X under the complex scenario j , obtain the test data in the complex scene and transfer the evaluation index in the minimum scene to the evaluation index in the complex scene by weight information entropy fusion, and obtain the weight W corresponding to the i-th minimum scene in the complex scene i , 1≤i≤I, complete the evaluation of the robot in complex scenarios.

[0010] Furthermore, the test data required for the evaluation index corresponding to the minimum scenario is obtained, the test data is labeled, and the evaluation index x under the minimum scenario is obtained using the parameter optimization model. ij The weight w ij , specifically including:

[0011] According to the set judgment rules, the test data and related calculation results directly and / or indirectly related to the evaluation indicators are marked, and then the weight w of each evaluation indicator in the minimum scenario is obtained through the parameter optimization model. ij .

[0012] Furthermore, the I minimum scenarios are coupled to obtain a complex scenario, and the evaluation index X under the complex scenario is set. j , specifically including:

[0013] Based on the robot evaluation task, the environmental parameters of the complex scene are defined on the basis of the minimum scene, and the complex scene is determined; the environmental parameters of the complex scene include all or part of the environmental parameters of the minimum scene; the evaluation index X under the complex scene j , including evaluation metrics for minimal scenarios;

[0014] By using the weight information entropy fusion method, the weight W of each minimum scene in the complex scene is determined. i ; Complete the evaluation of robots in complex scenarios.

[0015] Furthermore, the test data under the complex scene is obtained and the evaluation index under the minimum scene is transferred to the evaluation index under the complex scene by means of weight information entropy fusion, and the weight W corresponding to the i-th minimum scene in the complex scene is obtained. i, specifically including:

[0016] Based on the principle of information entropy invariance, the equation is established:

[0017] ∑w ij *h ij *W i =H j ;

[0018] Among them, h ij Represents the evaluation index x based on the minimum scenario ij The calculated information entropy is used to measure the uncertainty of data:

[0019] h ij =-∑(x ij )*log 2(x ij );

[0020] H j Represents the evaluation index X based on complex scenarios j Calculated information entropy;

[0021] Solve the equation to obtain the weight W of each minimum scene in the complex scene i .

[0022] Furthermore, the minimum scenario can meet the requirements for performance testing of a certain function of the robot, specifically including:

[0023] One minimum scenario can meet the requirements for performance testing of a certain function of the robot, or multiple minimum scenarios can meet the requirements for performance testing of a certain function of the robot.

[0024] Compared with the prior art, the beneficial technical effects of the present invention are:

[0025] The present invention completes robot evaluation through a minimum scenario, effectively improving the cost-effectiveness of robot evaluation and replacement. At the same time, in the minimum scenario, the influencing factors can be effectively controlled, improving the evaluation efficiency and accuracy. At the same time, based on the test in the minimum scenario, the consistency of the test results can be guaranteed, and the test data can be labeled according to certain judgment rules.

[0026] The present invention realizes the fusion of the robot's minimum scene and complex scene through the method of weighted information entropy fusion, obtains the evaluation result under the complex scene based on the minimum scene evaluation result, and solves the problem that the complex scene test data cannot be labeled according to certain judgment rules.

[0027] This paper designs and evaluates multiple minimal scenarios to comprehensively examine the robot's performance across different functions and environments. Furthermore, by combining these minimal scenarios to form complex scenarios, it can better simulate actual application scenarios and provide more accurate performance evaluations. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is an overall schematic diagram of the robot evaluation method of the present invention;

[0029] Figure 2 is a flow chart of the robot evaluation method of the present invention;

[0030] Figure 3a and Figure 3b They respectively represent the minimum scenario test evaluation process diagram and the complex scenario test evaluation process diagram of the present invention. DETAILED DESCRIPTION

[0031] A preferred embodiment of the present invention will be described in detail below with reference to the accompanying drawings.

[0032] This embodiment takes the evaluation of the autonomous navigation performance of a ground mobile robot as an example, and the robot evaluation method specifically includes the following steps:

[0033] S1, define the minimum scene environment parameters and determine the minimum scene:

[0034] Based on the autonomous navigation performance evaluation task for a ground mobile robot, the minimum scenario required for this evaluation task is defined, and the environmental parameters of the minimum scenario are determined. The environmental parameters of the minimum scenario include physical environment parameters, weather environment parameters, and interference environment parameters. Physical environment parameters include terrain parameters, spatial parameters, and obstacle parameters; weather environment parameters include lighting parameters, humidity parameters, and temperature parameters; and interference environment parameters include strong light parameters, smoke parameters, and dust parameters. This minimum scenario can cover the test requirements for the autonomous navigation performance of a ground mobile robot.

[0035] After clarifying the minimum scene environmental parameters, the minimum scene is determined based on the ground mobile robot autonomous navigation performance evaluation task. The minimum scene includes but is not limited to:

[0036] Component-level minimum test scenarios, i.e., minimum scenarios for devices and algorithms for autonomous navigation of ground mobile robots, such as minimum scenarios for lidar performance testing and perception algorithms;

[0037] Minimum interference scenarios include obstacle avoidance, straight path, constant curvature, variable curvature, and undulating terrain. The environmental complexity of these scenarios is consistent or similar.

[0038] Scenes with minimal interference such as scenes with minimal strong light interference and scenes with minimal weak light interference.

[0039] S2, clarify the minimum scenario evaluation indicators:

[0040] After determining the minimum scenario for the autonomous navigation performance test of the ground mobile robot, the evaluation index x of each minimum scenario is clarified. ij , x ij Used to measure the test performance of the robot's j-th function in the i-th minimum scenario.

[0041] In a preferred embodiment, the minimum scenario-related evaluation indicators (not limited to the following minimum scenarios and evaluation indicators) are as follows:

[0042] Minimum scenario evaluation indicators for component-level lidar performance testing include: lidar ranging capability, range accuracy, range precision, angular resolution, angular accuracy, angular accuracy, etc. Minimum scenario evaluation indicators for perception algorithms include: absolute trajectory error (ATE), relative pose error (RPE), etc.

[0043] The evaluation indicators for the minimum obstacle avoidance scenario include: the obstacle avoidance reaction distance of the ground mobile robot, the distance between the obstacle and the robot (when parallel to the obstacle), the distance away from the obstacle, the narrowest distance between two obstacles that can be passed, mileage, etc.; the minimum scenarios for straight lines, constant curvature, and variable curvature paths: ATE, RPE, completion time T, path smoothness, mileage, etc.; the minimum scenario for undulating terrain: the maximum slope that can be passed, the stability of the robot's posture, etc.

[0044] Evaluation indicators for scenes with minimal interference, such as scenes with minimal strong light interference and scenes with minimal weak light interference, include: map completion, positioning accuracy, etc.

[0045] S3, minimum scenario test, obtains the weight corresponding to the minimum scenario evaluation index:

[0046] Complete the establishment of the minimum scene in a simulation environment and / or a real environment, and obtain simulation test data / real test data that is directly and / or related to the evaluation indicators of each minimum scene obtained in step S2 through the test equipment; for example, obtain the minimum scene, mileage, and posture, speed, acceleration, etc. of a ground mobile robot completing a certain minimum scene.

[0047] Process the acquired data, that is, calculate the test data indirectly related to the evaluation indicators, and then label the processed data according to the minimum scene evaluation criteria, and determine the weights w of each evaluation indicator in the minimum scene through the trained parameter optimization model. ij , complete the evaluation of the minimum scenario.ij Indicates the evaluation index x in the minimum scenario ij The corresponding weight.

[0048] S4, minimum scenario coupling, determines complex scenarios:

[0049] Repeat the above steps to complete the evaluation of other minimum scenes. After completing the evaluation of each minimum scene, modify each minimum scene appropriately according to the evaluation results, such as the degree of strong light interference, maximum slope and other environmental parameters. Then, couple the minimum scenes to obtain a complex scene. The complex scene includes one or more complete or incomplete minimum environments determined in the first step, and can be randomly arranged in the complex scene, so that the coupled complex scene is more in line with the actual working conditions of the autonomous navigation of the ground mobile robot.

[0050] S5, clarify the evaluation indicators for complex scenarios:

[0051] After completing the complex scene determination, determine the evaluation index X of the ground mobile robot in the complex environment j (X j represents the jth evaluation indicator in a complex scenario), such as completion time T, absolute trajectory error, relative pose error, map completeness, mileage, obstacle avoidance reaction distance, etc. According to steps S3 and S4, simulation test data / real test data directly and / or indirectly related to the evaluation indicators in the complex environment are obtained, and the obtained data are processed, that is, the test data indirectly related to the evaluation indicators are calculated to obtain the data corresponding to the evaluation indicators.

[0052] S6, complex scenario test, determine the minimum weight of each scenario:

[0053] According to the weight information fusion algorithm, the weight W of each minimum scene in the complex scene is determined i , W i Indicates the weight corresponding to the i-th minimum scene in constructing a complex scene;

[0054] Specifically:

[0055] According to the information entropy calculation formula h ij =-∑(x ij )*log 2(x ij ) Determine the minimum scene evaluation index x contained in the complex scene ij The corresponding information entropy h ij Similarly, the evaluation index X of the complex environment is calculated j The corresponding information entropy H j , the minimum scene evaluation index x ij It has been determined in the minimum scenario evaluation; based on the principle of information entropy invariance, the equation ∑w is established ij*h ij *W i =H j , where H j With h ij The evaluation index X of the same robot function in complex scenes and minimum scenes respectively j and x ij The corresponding information entropy; w ij is the evaluation index x in the minimum scenario ij According to the information entropy of each evaluation index determined by the complex scene and the minimum scene, the proportion of each minimum scene in the complex scene is obtained. i , and based on the minimum scene evaluation judgment rules, the test of the autonomous navigation performance of the ground mobile robot in complex scenes is obtained.

[0056] S7, complete the complex scenario test evaluation based on the minimum scenario.

[0057] In summary, the present invention first defines the minimum scene environment parameters, and the minimum scene can meet the requirements of a key function of the robot; secondly, the evaluation indicators under the minimum scene are set, and the evaluation indicators under the minimum scene measure a certain performance of the robot; then, the robot is tested on the basis of the minimum scene, and the relevant test data required for the indicators corresponding to the minimum scene are obtained, the test data is labeled, and the parameter optimization model is used to obtain the weights of each evaluation indicator in the minimum scene, and the evaluation of the robot by the minimum scene is completed; finally, on the basis of completing multiple minimum scenes, multiple minimum scenes are coupled to obtain a complex scene and the evaluation indicators under the complex scene are set, and the evaluation indicators under the minimum scene are migrated to the evaluation indicators under the complex scene by means of weight information entropy fusion, and the weights of each minimum scene in the complex scene are obtained, thereby completing the test and evaluation of the robot in the complex scene.

[0058] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. It is intended that all variations within the meaning and range of equivalents of the claims be embraced herein, and any reference signs in the claims should not be construed as limiting the claims to which they relate.

[0059] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A robot evaluation method based on a minimum scenario, characterized in that: include: Define the minimum scene environment parameters based on the robot evaluation tasks and test standards, and complete the construction of the minimum scene; The minimum scenario can meet the requirements for performance testing of a certain function of the robot; Set the evaluation index x under the minimum scenario ij , the evaluation index x in the minimum scenario ij It is used to measure the test performance of the jth function of the robot in the i-th minimum scenario, 1≤j≤J, J represents the total number of functions of the robot; the robot is tested on the basis of the minimum scenario, the relevant test data required for the evaluation index corresponding to the minimum scenario is obtained, the test data is annotated, and the evaluation index x under the minimum scenario is obtained using the parameter optimization model ij The weight w ij , complete the evaluation of the robot in the minimum scenario; Couple I minimum scenarios to obtain a complex scenario, and set the evaluation index X under the complex scenario j , obtain the test data in the complex scene and transfer the evaluation index in the minimum scene to the evaluation index in the complex scene by weight information entropy fusion, and obtain the weight W corresponding to the i-th minimum scene in the complex scene i , 1≤i≤I, complete the evaluation of the robot in complex scenarios.

2. The robot evaluation method based on the minimum scenario according to claim 1, characterized in that: The method is to obtain the test data required for the evaluation index corresponding to the minimum scenario, mark the test data, and use the parameter optimization model to obtain the evaluation index x under the minimum scenario. ij The weight w ij , specifically including: According to the set judgment rules, the test data and related calculation results directly and / or indirectly related to the evaluation indicators are marked, and then the weight w of each evaluation indicator in the minimum scenario is obtained through the parameter optimization model. ij .

3. The robot evaluation method based on the minimum scenario according to claim 1, characterized in that: The I minimum scenarios are coupled to obtain a complex scenario, and the evaluation index X is set under the complex scenario. j , specifically including: Based on the robot evaluation task, the environmental parameters of the complex scene are defined on the basis of the minimum scene, and the complex scene is determined; the environmental parameters of the complex scene include all or part of the environmental parameters of the minimum scene; the evaluation index X under the complex scene j , including evaluation metrics for minimal scenarios; By using the weight information entropy fusion method, the weight W of each minimum scene in the complex scene is determined. i ; Complete the evaluation of robots in complex scenarios.

4. The robot evaluation method based on the minimum scenario according to claim 1, characterized in that: The test data under the complex scene is obtained and the evaluation index under the minimum scene is transferred to the evaluation index under the complex scene by weight information entropy fusion, and the weight W corresponding to the i-th minimum scene in the complex scene is obtained. i , specifically including: Based on the principle of constant information entropy, the equation is established: ∑w ij *h ij *W i =H j ; Among them, h ij Represents the evaluation index x based on the minimum scenario ij The calculated information entropy is used to measure the uncertainty of data: h ij =-∑(x ij )*log2(x ij ); H j Represents the evaluation index X based on complex scenarios j Calculated information entropy; Solve the equation to obtain the weight W of each minimum scene in the complex scene i .

5. The robot evaluation method based on a minimum scenario according to claim 1, characterized in that: The minimum scenario can meet the requirements for performance testing of a certain function of the robot, including: One minimum scenario can meet the requirements for performance testing of a certain function of the robot, or multiple minimum scenarios can meet the requirements for performance testing of a certain function of the robot.

Images