Automatic Evaluation Method and System for Performance Testing of Human-Robot Interaction Service Robots

Abstract

The present invention provides an automatic evaluation method and system for the performance test of a human-computer interaction service robot, including: obtaining information about obstacles around the robot and information about the robot itself; inputting starting point and target point data for the robot and a navigation method to be tested to obtain a navigation trajectory; calculating the average travel time of the robot in the most comfortable space for the user, the average duration of intersection between the predicted projection path of the robot and the predicted projection path of a pedestrian, the path length ratio, the path irregularity, the trajectory smoothness, the robot movement speed, the left-right side preference, the path length, and the travel time based on the obtained information and the navigation trajectory; evaluating the service robot using different navigation methods in a specific scenario from the aspects of efficiency, comfort, naturalness, and sociality based on the calculated index data.

Description

Technical Field

[0001] The present invention belongs to the technical field of mobile robot navigation, and particularly relates to an automatic evaluation method and system for the performance test of a human-computer interaction service robot. Background Art

[0002] The statements in this part only provide background technical information related to the present invention, and do not necessarily constitute prior art.

[0003] With the rapid development of computer technology, service robots have also entered people's daily lives. The positioning of service robots determines an important characteristic of their working environment - human-robot coexistence. Facing the complex dynamic environment of human-robot coexistence, higher requirements are put forward for the navigation planning tasks of robots. On the one hand, it is necessary to ensure the safety of users on the premise of completing the work tasks. On the other hand, it is required that the robot consider the comfort of users during the work process, not hinder the activities of users, and at the same time, the behavior of the robot should conform to the behavior norms of human society.

[0004] In recent years, the progress of computer hardware and machine learning has enabled the implementation of a series of social navigation algorithms based on deep reinforcement learning. By encoding social behaviors that conform to human behavior norms and models considering user comfort in the reward function, and then providing rewards or punishments to the robot according to its behavior, guiding the robot to complete the social navigation planning task. Therefore, how to select a navigation planning algorithm that matches the actual scenario has become an important issue that researchers urgently need to solve.

[0005] Evaluating the advantages and disadvantages of a navigation algorithm is determined by a series of indicators. However, in order to prove the superiority of the algorithms they proposed, researchers often construct evaluation indicators based on their navigation methods. Since people have different definitions of "social behavior", the indicators for training and evaluating navigation methods are also different. Without unified indicators, it is impossible to compare the advantages and disadvantages of different navigation algorithms, and thus select the most suitable navigation algorithm for the current scenario.

[0006] Currently, almost all evaluations of navigation methods consistently report task-related indicators, such as success rate or time to reach the target point, but rarely mention quantitative indicators related to human-computer interaction. Although efficiency is an important aspect of social navigation, with the rapid development of social economy, people's requirements for a comfortable life are gradually increasing. In the complex dynamic environment of human-robot coexistence, how to improve user comfort on the premise of ensuring user safety has become an important indicator for evaluating navigation methods. Summary of the Invention

[0007] To overcome the deficiencies of the above-mentioned prior art, the present invention provides an automatic evaluation method for the performance test of a human-computer interaction service robot. Starting from four aspects: efficiency, comfort, naturalness, and sociality, the autonomous navigation method based on social behavior modeling can quantify the evaluation system, reflecting the advantages and disadvantages of different navigation algorithms, so as to select a suitable navigation method for different scenarios.

[0008] To achieve the above object, one or more embodiments of the present invention provide the following technical solutions:

[0009] In the first aspect, an automatic evaluation method for the performance test of a human-computer interaction service robot is disclosed, including:

[0010] Obtain the information of the obstacles around the robot and the information of the robot itself;

[0011] Input the starting point and target point data of the robot and the navigation method to be tested to obtain the navigation trajectory;

[0012] Based on the obtained information and the navigation trajectory, calculate the average travel time of the robot in the most comfortable space of the user, the average duration of the intersection of the predicted projection path of the robot and the predicted projection path of the pedestrian, the path length ratio, the path irregularity, the trajectory smoothness, the movement speed of the robot, the left-right preference, the path length, and the travel time;

[0013] Based on the calculated index data, evaluate the service robot using different navigation methods in a specific scenario from the aspects of efficiency, comfort, naturalness, and sociality.

[0014] As a further technical solution, the average travel time of the robot in the most comfortable space of the user and the average duration of the intersection of the predicted projection path of the robot and the predicted projection path of the pedestrian are used as indicators of the comfort of the navigation algorithm;

[0015] The path length ratio, the path irregularity, the trajectory smoothness, and the movement speed of the robot are used as indicators of the naturalness of the navigation algorithm;

[0016] The left-right preference is used as an indicator of the sociality of the navigation algorithm;

[0017] The path length and the travel time are used as indicators of the efficiency of the navigation algorithm.

[0018] As a further technical solution, the information of the obstacles around the robot includes the position information and speed information of the obstacles around the robot; the information of the robot itself includes the speed and position information of the robot itself.

[0019] As a further technical solution, the smaller the average travel time of the robot in the most comfortable space of the user, the stronger the ability of the robot to respect the personal space of the user.

[0020] As a further technical solution, the smaller the average duration of the intersection between the predicted projection path of the robot and the predicted projection path of the pedestrian, the stronger the ability of the robot to avoid collisions.

[0021] As a further technical solution, the path length ratio describes the relationship between the optimal trajectory length from the starting point to the destination and the length of the actual navigation route, and is the ratio of the travel length of the simulation trajectory based on the SFM algorithm to the length of the actual navigation route.

[0022] As a further technical solution, when calculating the path irregularity:

[0023] Establish a rectangular coordinate system: Take the starting point as the origin, the straight line where the ray from the starting point to the destination is located as the y-axis, and the x-axis is the straight line passing through the origin and perpendicular to the y-axis;

[0024] Point sampling: Let the path length be l, take M points, and sample one point every Sample one point along the path;

[0025] Calculate the velocity offset angle between adjacent points;

[0026] Sum up and average the velocity offset angles between all adjacent points, which is the path irregularity.

[0027] In a second aspect, an automatic evaluation system for the performance test of a human-computer interaction service robot is disclosed, including:

[0028] An information acquisition module, configured to: acquire information about obstacles around the robot and information about the robot itself;

[0029] For the robot, input the starting point and target point data and the navigation method to be tested to obtain a navigation trajectory;

[0030] An index calculation module, configured to: calculate the average travel time of the robot in the user's most comfortable space, the average duration of the intersection between the predicted projection path of the robot and the predicted projection path of the pedestrian, the path length ratio, the path irregularity, the trajectory smoothness, the robot's movement speed, the left-right side preference, the path length, and the travel time based on the acquired information and the navigation trajectory;

[0031] An evaluation module, configured to: evaluate the service robot in specific scenarios using different navigation methods from the aspects of efficiency, comfort, naturalness, and sociality based on the calculated index data.

[0032] As a further technical solution, the evaluation module includes:

[0033] Construct a hierarchical structure model, which includes an objective layer, a criterion layer, and a scheme layer;

[0034] Three judgment matrices of the hierarchical analysis method were established, namely based on emergency tasks, based on commercial use, and based on home use;

[0035] The horizontal and vertical coordinates of the judgment matrix are 9 indicators, and each element in the matrix is represented by a ij It represents the importance of i to j, and the subjective weight is determined according to the judgment matrix;

[0036] The standardized matrix is further processed to obtain the characteristic weight matrix;

[0037] Calculate the weight matrix;

[0038] Combine the subjective weight with the objective weight to calculate the combined weight;

[0039] Get an overall evaluation of different navigation methods, as well as corresponding evaluations of efficiency, comfort, naturalness, and sociability.

[0040] One or more of the above technical solutions have the following beneficial effects:

[0041] During navigation performance testing, the present invention combines several robot-mounted sensors to collect environmental information surrounding the robot, and utilizes the robot operating system to communicate with the robot in real-world scenarios or simulated test environments, thereby achieving automatic quantitative evaluation of different modules. Specifically, based on the four modules of comfort, naturalness, sociability, and efficiency, nine evaluation indicators are proposed by modeling social behavior to form an evaluation scale. The real-time data obtained from the scale is then secondary optimized using the AHP-entropy weight method and displayed using a visual interface. This present invention enables navigation performance testing of human-machine interactive service robots from multiple perspectives, comprehensively and automatically evaluating the performance of navigation algorithms based on fixed navigation scenarios, and provides guidance for optimizing navigation algorithms.

[0042] The present invention proposes a unified evaluation index for evaluating the pros and cons of navigation methods, and can quantify the performance of the navigation methods on the index, so as to select the most appropriate navigation method in a fixed scenario.

[0043] In the complex dynamic environment of human-machine integration, in order to improve user comfort while ensuring user safety, the present invention proposes concepts such as speed rectangle and social models such as left-right preference to quantify the evaluation of sociability.

[0044] Based on the decoupling process, the performance of different modules of the navigation algorithm can be tested, so that targeted guidance can be provided.

[0045] Advantages of additional aspects of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0047] Figure 1 Flowchart of the automatic evaluation system designed by the present invention for testing the navigation performance of a human-machine interactive service robot;

[0048] Figure 2 A conceptual diagram of the speed rectangle proposed by the present invention;

[0049] Figure 3 Schematic diagrams of three common interaction scenarios in daily life used in the present invention;

[0050] Figure 4 An evaluation index system for a quantifiable evaluation system of the autonomous navigation method of the present invention;

[0051] Figure 5 This is a schematic diagram of the example effect;

[0052] Figure 6 Schematic diagram of AHP-entropy weight method. DETAILED DESCRIPTION

[0053] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0054] It should be noted that the terms used herein are for describing particular embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.

[0055] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0056] Example 1

[0057] This embodiment discloses an automatic evaluation method for performance testing of a human-machine interactive service robot. Figure 1 As shown, the following steps are included:

[0058] Step 1: Information collection steps:

[0059] The robot's built-in sensors collect environmental information, including the position and velocity of obstacles around it, as well as the robot's own velocity and position. By inputting the starting and target points and the navigation method to be tested, a navigation trajectory is generated. To calculate the velocity offset angle, the tangent line at the sampling point on the trajectory represents the direction of the velocity vector.

[0060] Step 2. Data processing step:

[0061] The data obtained through information collection will sort and filter the initial data required for model calculation. See the model calculation steps for details:

[0062] Comfort-related indicators:

[0063] Comfort means that humans will not feel annoyed and stressed during the human-machine interaction process. Based on this characteristic, there are two evaluation indicators.

[0064] (1) The average travel time E1 of the robot in the user's most comfortable space. The smaller E1 is, the stronger the robot's ability to respect the user's personal space. Hall proposed the space division near pedestrians: the intimate distance is 0 - 0.45m; the personal distance is 0.45 - 1.2m; the social distance is 1.2 - 3.6m; more than 3.6m is the public distance; the distance is divided according to a percentage system. Considering that the closer the robot is to the pedestrian, the greater the impact of the time within this range on the pedestrian, different weight coefficients are assigned to the intimate distance, personal distance, and social distance according to an inverse proportion relationship.

[0065] a + b + c = 1 (1)

[0066] The weighted coefficient within the intimate distance is a = 0.67, the weighted coefficient within the personal distance is b = 0.21, and the weighted coefficient within the social distance is c = 0.12.

[0067]

[0068] N represents the total number of pedestrians entering the social distance of the pedestrian during the robot's movement, T represents the total navigation time of the robot, H represents the set of pedestrians, h represents the target pedestrian, r represents the robot, represents the position of the pedestrian at time t, represents the position of the robot at time t. Among them, the definition formula of N is as follows:

[0069]

[0070] During specific evaluation, T is obtained by the timing module of the robot, and the set of pedestrians H is obtained using the pedestrian detection algorithm YOLOV3.

[0071] (2) The average duration E2 of the intersection of the predicted projection path of the robot and the predicted projection path of the pedestrian. The smaller E2 is, the stronger the robot's ability to avoid collisions. First, a concept of a velocity rectangle is proposed. As Figure 2 shown, in the top view, the robot is regarded as a semi-circle, and a velocity rectangle is generated in its velocity direction, with a length of The width is the diameter of the semi - circle. The movement of pedestrians can also be regarded as a velocity rectangle. When the velocity rectangle of the robot intersects with that of the pedestrian, there is a risk of collision. The larger the intersecting area, the higher the risk.

[0072]

[0073] As Figure 2 shown, due to inertia, if the robot and the pedestrian move too fast, they are prone to collision. To simulate the degree of collision, a velocity rectangle is used to represent the influence range of the pedestrian or the robot in terms of its velocity. If there is an intersection between the two velocity rectangles, it indicates the existence of a collision risk. E2 calculates the average time during which the velocity rectangles intersect. represents the velocity rectangle of the robot, represents the velocity rectangle of the pedestrian.

[0074] Natural - related indicators:

[0075] Naturalness refers to the similarity between the behavior patterns of the robot and humans. Based on this characteristic, the proposed naturalness scoring module includes four evaluation indicators.

[0076] (1) Path - length ratio E3, which describes the relationship between the length of the optimal trajectory from the starting point to the destination and the length of the actual navigation route. Here, for the optimal trajectory, we selected the simulation trajectory based on the SFM algorithm. In the Socnavbench article, when comparing with currently relatively mature navigation algorithms, the SFM has the best effect. The larger E3 is, the more direct the path the robot takes.

[0077]

[0078] l d represents the travel length of the simulation trajectory based on the SFM algorithm, l s represents the length of the actual navigation route.

[0079] In this embodiment, the travel length of the simulation trajectory calculated based on the SFM algorithm. [[ID=_{35}]]

[0080] (2) Path irregularity E4, which reflects the degree of curvature of the navigation path. The smaller E4 is, the smaller the degree of curvature and the more direct the path. If it reaches the destination directly, then the path irregularity E4 = 0. The following is the calculation method of the path irregularity:

[0081] 2 - 1 Establish a rectangular coordinate system

[0082] Based on the navigation route map, establish a two - dimensional rectangular coordinate system. Take the starting point as the origin, the straight line where the ray from the starting point to the destination is located as the y - axis, and the x - axis is the straight line passing through the origin and perpendicular to the y - axis.

[0083] 2 - 2 point sampling

[0084] If the path length is l and M points are taken, one point is sampled every point along the path;

[0085] 2 - 3 Calculate the velocity deviation angle θ between adjacent points,

[0086]

[0087] In this embodiment, a total of M sampling points are selected, and each point is represented by a lowercase letter (1, 2, …, m), refers to the velocity vector at the m - th sampling point, with the magnitude being the velocity and the direction being the tangent to the trajectory at that point; refers to the magnitude of the velocity.

[0088] 2 - 4 Summing up and averaging all the velocity deviation angles θ between adjacent points gives the path irregularity, and the calculation formula is as follows:

[0089]

[0090] The E4 values are all greater than 0. The closer to 0, the smaller the bending degree and the straighter the path.

[0091] (3) Trajectory smoothness E5, which describes the smoothness of the robot's navigation trajectory. The larger E5 is, the smoother the trajectory and the better the effect.

[0092]

[0093] T represents the total time of the robot's navigation, and x(t) represents the displacement at the unit time t.

[0094] (4) Robot movement speed E6, which describes the average duration that the robot's speed exceeds the maximum normal walking speed of humans. The smaller E6 is, the better the effect. Since the maximum walking speed of humans is about 1.2 - 1.8 m / s, we take the average value here.

[0095]

[0096] Social - related indicators:

[0097] Sociality refers to the robot's compliance with clear high - level social habits, such as overtaking, passing by, crossing, etc. from a specific side. Based on this characteristic, an indicator including left - right side preference is included. To effectively test this indicator, we will select three relatively common interaction scenarios in daily life, such as Figure 3 shown.

[0098] The leftmost one is the "passing by" scenario, indicating that the pedestrian and the robot are walking towards each other, and after passing by each other, they continue to move forward. The second one from the left is the "overtaking" scenario, where the robot starts behind a pedestrian walking in the same direction (with a slower speed) and overtakes the pedestrian. The rightmost one is the "crossing" scenario, where the robot moves perpendicular to the direction of the pedestrian's advance, and their paths cross each other.

[0099] When the Euclidean distance between the robot and the pedestrian is less than a certain threshold, we compare their horizontal and vertical coordinates:

[0100] When in the "passing by" scenario on the leftmost side, for the sake of improving unity, modifications are made here. Regarding the left and right sides moving towards each other, the pedestrian is taken as the reference. If then the output is "left" of the pedestrian. If then the output is "right" of the pedestrian, where represents the x - coordinate of the position where the robot is located, represents the x - coordinate of the position where the pedestrian is located.

[0101] When in the "overtaking" scenario on the second left side, if then the output is "left", if then the output is "right", where represents the x - coordinate of the position where the robot is located, represents the x - coordinate of the position where the pedestrian is located.

[0102] When in the "crossing" scenario on the rightmost side, if then the output is "left", indicating that the robot will give way to the pedestrian to pass first; if then the output is "right", indicating that the robot will pass in front of the pedestrian in advance. Where represents the y - coordinate of the position where the robot is located, represents the y - coordinate of the position where the pedestrian is located.

[0103] Based on social etiquette and behavior habits, the left - right identification and analysis of the above three behaviors are carried out. When identified as left, it is more in line with social rules. The proportion of "left" in the scenario is statistically obtained as the output, denoted by E7.

[0104] Efficiency - related indicators:

[0105] Two indicators, the path length E8 and the travel time E9, are selected, which are respectively the path length that the robot walks to complete the navigation task and the time consumed.

[0106] Step 3: Score output step:

[0107] The present invention constructs a quantifiable evaluation system with the comfort, naturalness, sociality, and efficiency of the navigation algorithm as the first-level indicators, and these three first-level indicators further include the above-mentioned 9 secondary indicators.

[0108] The specific technical solution is as follows:

[0109] The first-level indicators include four indicators: efficiency, comfort, naturalness, and sociality, which are greatly affected by people's subjective feelings. Therefore, in combination with the form of a questionnaire survey, using the analytic hierarchy process to assign weight coefficients can better highlight the requirements for different indicators in special scenarios. The subjective weight design process is as follows:

[0110] (1) Establish a hierarchical structure model, as shown in the appendix Figure 4 The hierarchical structure model includes an objective layer, a criterion layer, and a scheme layer.

[0111] (2) Since different task scenarios have different requirements for navigation performance. For example, emergency tasks such as delivering medicine have obvious requirements for efficiency, while service robots in the home environment tend to be more comfortable. Therefore, based on the results of the questionnaire survey, three judgment matrices of the analytic hierarchy process are proposed, and all of them have passed the consistency test, namely, based on emergency tasks, based on commercial use, and based on home use. The judgment matrices are as follows:

[0112]

[0113] Judgment matrix based on emergency tasks

[0114]

[0115] Judgment matrix based on commercial use

[0116]

[0117] Judgment matrix based on home use

[0118] The horizontal and vertical coordinates of the judgment matrix are 9 indicators respectively, and each element in the matrix is represented by a ij indicating the importance degree of i relative to j. Determine the subjective weight W (a)j .

[0119] Subjective weight matrix: W (a) =[w (a)1 w (a)2 … w (a)i [[ID=�2]]

[0120]

[0121] The objective weight design process is as follows:

[0122] (1) Construct an index matrix​ There are m evaluation objects and n evaluation indicators (n = 9). In the formula, X ij is the original data of the i-th evaluation object under the j-th indicator.

[0123] (2) Perform dimensionless processing on the original data to obtain the standardized matrix V.

[0124]

[0125] In the formula, v ij is the standardized value of the i-th evaluation object under the j-th indicator.

[0126] For the three positive indicators E3, E5, and E7, the calculation method of v ij is as follows:

[0127]

[0128] In the formula, max(x j ) and min(x j ) are the maximum and minimum values of all evaluation objects under the j-th indicator respectively.

[0129] For the six negative indicators E1, E2, E4, E6, E8, and E9, the calculation method of v ij is as follows:

[0130]

[0131] (3) Further process the standardized matrix to obtain the feature proportion matrix P:

[0132]

[0133] In the formula, p ij is the feature proportion of the i-th evaluation object under the j-th indicator, and the calculation method is as follows:

[0134]

[0135] Calculate the entropy value matrix E as follows:

[0136] E = [e1 e2 … e n

[0137] In the formula, e j is the entropy value of the j-th indicator, and the calculation method is as follows:

[0138]

[0139] (5) Calculate the weight matrix as follows

[0140] W​(e) = [w (e)1 w (e)2 … w (e)j

[0141] Among them,

[0142] By combining the subjective weight and the objective weight, the combined weight calculation formula is:

[0143] Perform data processing on the evaluation indicators. Z represents the scoring matrix, V represents the standardized matrix, ω represents the weight matrix, and the final score calculation formula is:

[0144]

[0145] According to the final score calculation formula, the score of the comfort module is The score of the naturalness module is The score of the sociality module is Z soc = Z7, and the score of the efficiency module is In this way, not only can the overall score Z of different navigation methods be obtained, but also the scores of different modules can be obtained, Figure 5 which is the visualization interface. It provides a targeted direction for the optimization of subsequent navigation methods.

[0146] Embodiment 2

[0147] The purpose of this embodiment is to provide a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps of the above method are implemented.

[0148] Embodiment 3

[0149] The purpose of this embodiment is to provide a computer-readable storage medium.

[0150] A computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the steps of the above method are executed.

[0151] Embodiment 4

[0152] The purpose of this embodiment is to provide an automatic evaluation system for the performance test of a human-computer interaction service robot, including:

[0153] An information acquisition module, configured to: acquire information about obstacles around the robot and information about the robot itself;

[0154] For the robot, input the starting point and target point data and the navigation method to be tested to obtain a navigation trajectory;

[0155] ​An index calculation module, configured to: calculate the average travel time of the robot in the user's most comfortable space, the average duration of the intersection between the predicted projection path of the robot and the predicted projection path of the pedestrian, the path length ratio, the path irregularity, the trajectory smoothness, the robot movement speed, the left - right side preference, the path length, and the travel time based on the acquired information and the navigation trajectory;

[0156] An evaluation module, configured to: evaluate the service robot in a specific scenario using different navigation methods respectively from the aspects of efficiency, comfort, naturalness, and sociality based on the calculated index data.

[0157] By modeling social behavior, nine evaluation indexes are proposed to form an evaluation scale. Based on the AHP - entropy weight method, the real - time data obtained from the scale is optimized twice, and a visualization interface is used for display. The present invention enables the test of the human - machine interaction service robot to comprehensively and automatically evaluate the navigation performance from multiple perspectives, reduces manual intervention, lowers costs, and speeds up the test. Due to the decoupling process, the performance of different modules of the navigation algorithm can be detected, so targeted guiding directions can be proposed for it.

[0158] Specifically, the robot sensors include lidar, millimeter - wave radar, cameras, IMU, encoders, etc.

[0159] The environmental information includes the position information, speed information, visual classification information of the obstacles around the robot, the starting - point position information, and the target - point position information.

[0160] The communication between ROS and the robot in the real - world scenario or the simulation test environment specifically includes: in the real - world scenario, writing a program based on the ROS platform using python and implementing the websocket function using the socket library, so that the sensors of the robot can send navigation parameters to the client computer through the TCP protocol; in the simulation test environment, writing a program based on the ROS platform using C++, and obtaining the navigation parameters of the simulation software through the information receiving and sending mechanism based on the MQTT protocol in ROS and sending them to the client for calculation.

[0161] The different modules include a comfort module, a safety module, a sociality module, and an efficiency module. Each evaluation module includes different evaluation indexes, representing different characteristics under different modules, thus forming a scale. When formulating the indexes, the influence on the completion degree of the navigation task, human psychology, and sociology is fully considered, and each index item of the module is ensured to be decoupled, which is convenient for subsequent use and judgment.

[0162] The automatic evaluation is mainly based on the AHP - entropy weight method, as Figure 6 shown.

[0163] The input is the original data of different navigation algorithms based on scales. Due to the limitations of a single weighting method, the analytic hierarchy process (AHP) and the entropy weight method are combined. When calculating the subjective weight, three judgment matrices of the AHP are proposed according to the results of the questionnaire survey, and all of them pass the consistency test, namely, based on emergency tasks, based on commercial use, and based on household use. For the output, not only can the final score of the navigation algorithm be obtained, but the different modules are also decoupled to obtain the specific scores of the corresponding modules.

[0164] In the devices of the above second, third, and fourth embodiments, the steps involved correspond to those in the first method embodiment. For the specific implementation manner, reference may be made to the relevant description part of the first embodiment. The term "computer-readable storage medium" should be understood to include a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode, or carry an instruction set for execution by a processor and cause the processor to execute any method in the present invention.

[0165] Those skilled in the art should understand that the above-mentioned modules or steps of the present invention can be implemented by a general-purpose computer device. Optionally, they can be implemented by program codes executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be separately fabricated into individual integrated circuit modules, or multiple modules or steps among them can be fabricated into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0166] Although the specific implementation manners of the present invention are described above in conjunction with the drawings, it is not a limitation to the protection scope of the present invention. Those skilled in the art should understand that based on the technical solutions of the present invention, various modifications or deformations that can be made without creative efforts by those skilled in the art are still within the protection scope of the present invention.

Claims

1. An automatic evaluation method for the performance test of a human-computer interaction service robot, characterized in that, Including: Obtaining information about obstacles around the robot and information about the robot itself; Inputting starting point and target point data for the robot and the navigation method to be tested to obtain a navigation trajectory; Based on the obtained information and navigation trajectory, calculating the average travel time of the robot in the most comfortable space for the user, the average duration of the intersection of the predicted projection path of the robot and the predicted projection path of the pedestrian, the path length ratio, the path irregularity, the trajectory smoothness, the movement speed of the robot, the left-right preference, the path length, and the travel time; Evaluating the service robot using different navigation methods in a specific scenario from the aspects of efficiency, comfort, naturalness, and sociality based on the calculated index data; The evaluating the service robot using different navigation methods in a specific scenario from the aspects of efficiency, comfort, naturalness, and sociality based on the calculated index data includes: Constructing a hierarchical structure model, which includes an objective layer, a criterion layer, and a scheme layer; Establishing three judgment matrices for the analytic hierarchy process, namely, based on emergency tasks, based on commercial use, and based on household use; The horizontal and vertical coordinates of the judgment matrix are 9 indicators respectively, and each element in the matrix is represented by which represents equivalent to importance degree. The subjective weight is determined according to the judgment matrix; Construct an index matrix , there are m evaluation objects and n evaluation indicators, is the original data of the m-th evaluation object under the th indicator; Performing dimensionless processing on the original data to obtain a standardized matrix; Further processing the standardized matrix to obtain a characteristic proportion matrix; Calculating an entropy value matrix; Calculating an objective weight matrix; Combining the subjective weight and the objective weight to calculate a combined weight; Obtaining the overall evaluation of different navigation methods and obtaining the evaluations corresponding to efficiency, comfort, naturalness, and sociality.

2. The automatic evaluation method for the performance test of a human-computer interaction service robot according to claim 1, characterized in that, The average travel time of the robot in the most comfortable space for the user and the average duration of the intersection of the predicted projection path of the robot and the predicted projection path of the pedestrian are used as indicators of the comfort of the navigation algorithm; The path length ratio, the path irregularity, the trajectory smoothness, and the movement speed of the robot are used as indicators of the naturalness of the navigation algorithm; The left-right preference is used as an indicator of the sociality of the navigation algorithm; The path length and the travel time are used as indicators of the efficiency of the navigation algorithm.

3. The automatic evaluation method for the performance test of a human-computer interaction service robot according to claim 1, characterized in that, The information about obstacles around the robot includes the position information and speed information of obstacles around the robot; the information about the robot itself includes the speed and position information of the robot itself.

4. The automatic evaluation method for the performance test of a human-computer interaction service robot according to claim 1, characterized in that, The path length ratio describes the relationship between the optimal trajectory length from the starting point to the destination and the actual navigation route length, and is the ratio of the travel length of the simulation trajectory based on the SFM algorithm to the actual navigation route length.

5. The automatic evaluation method for the performance test of a human-computer interaction service robot according to claim 1, characterized in that, When calculating the path irregularity: Establishing a rectangular coordinate system: taking the starting point as the origin, the straight line where the ray pointing from the starting point to the destination is located as the y-axis, and the x-axis is the straight line passing through the origin and perpendicular to the y-axis; Point sampling: Let the path length be , and take points, sampling one point every of the path; Calculating the speed deviation angle between adjacent points; Summing up and averaging the speed deviation angles between all adjacent points to obtain the path irregularity.

6. An automatic evaluation system for the performance test of a human-computer interaction service robot, characterized in that Including: An information acquisition module configured to: obtain information about obstacles around the robot and information about the robot itself; Inputting starting point and target point data for the robot and the navigation method to be tested to obtain a navigation trajectory; An indicator calculation module is configured to: calculate, based on the acquired information and the navigation trajectory, an average travel time of the robot in the user's most comfortable space, an average duration of intersection between the robot's predicted projected path and the pedestrian's predicted projected path, a path length ratio, path irregularity, trajectory smoothness, robot movement speed, left-right preference, path length, and travel time; The evaluation module is configured to: evaluate the efficiency, comfort, naturalness, and sociability of different navigation methods used by the service robot in specific scenarios based on the calculated indicator data; Assessment modules include: Constructing a hierarchical model, wherein the hierarchical model includes a target layer, a criterion layer, and a solution layer; Three judgment matrices of the hierarchical analysis method were established, namely based on emergency tasks, based on commercial use, and based on home use; The horizontal and vertical coordinates of the judgment matrix are 9 indicators respectively, and each element in the matrix is represented by indicating equivalent to importance degree. The subjective weight is determined according to the judgment matrix; Construct an index matrix , there are m evaluation objects and n evaluation indicators, is the original data of the th evaluation object under the th indicator; Perform dimensionless processing on the original data and obtain the standardized matrix; The standardized matrix is further processed to obtain the characteristic weight matrix; Calculate the entropy matrix; Calculate the objective weight matrix; Combine the subjective weight with the objective weight to calculate the combined weight; Get an overall evaluation of different navigation methods, as well as corresponding evaluations of efficiency, comfort, naturalness, and sociability.

7. The automatic evaluation system for the performance test of a human-computer interaction service robot according to claim 6, characterized in that, The robot operating system communicates with the robot in a real scene or a simulated test environment. Specifically, the robot's built-in sensors send navigation parameters to the client computer through the TCP protocol; the navigation parameters of the simulation software are obtained through the information sending and receiving mechanism based on the MQTT protocol in the operating system, and sent to the client for calculation.

8. A computer device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, the steps of the method described in any one of claims 1 to 5 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, the steps of the method described in any one of claims 1 to 5 are executed.

Images