A method for evaluating human-machine collaborative agents

Abstract

The present invention discloses an evaluation method for a human-machine collaborative intelligent body, which relates to the technical field of intelligent body evaluation. By analyzing the power A1, ripple voltage A2, sending time A3, TP address A4, byte length A5, and receiving time A6 of data A, the interference value formula of the data is used to timely obtain the interference degree of the data, and on this basis, the abnormal value of the data is further calculated, and the data that is seriously interfered with is timely re-collected, and the data A that suffers from data poisoning is timely corrected through a data correction module, and the data of the complete time series is sent to a result evaluation module to ensure the consistency and accuracy of the data, thereby improving the accuracy and reliability of the evaluation results of the human-machine collaborative intelligent body.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent agent evaluation, and in particular to an evaluation method for a human-machine collaborative intelligent agent. Background Art

[0002] The evaluation of human-machine collaborative agents refers to the process of evaluating and testing the performance, efficiency, quality, stability, adaptability and other aspects of the collaborative work between humans and machines (agents). This evaluation method aims to measure the overall performance of human-machine collaborative agents to ensure that they can meet the expected goals in practical applications and provide users with efficient, accurate and reliable services.

[0003] The key factor in the evaluation of human-machine collaborative agents lies in the quality of data. In the process of processing information, human-machine collaborative agents are extremely dependent on the accuracy of input data. If there is environmental interference or "data poisoning" during the evaluation process, errors, noise or inconsistencies will appear in the data, which will greatly affect the accuracy and reliability of the evaluation results of human-machine collaborative agents. Summary of the invention

[0004] In response to the above situation, the present invention can accurately detect the evaluation data in real time, and promptly re-collect the data that is severely interfered with, and promptly correct the data that is poisoned, so as to ensure the consistency and accuracy of the data, thereby improving the accuracy and reliability of the evaluation results of the human-machine collaborative intelligent body.

[0005] The technical solution is: S1, setting the human-machine collaborative intelligent body decision-making task with preset environmental parameters and preset task difficulty through the task setting module. During the test, the data acquisition module collects data A of the human-machine collaborative intelligent body in real time;

[0006] S2, a, the data analysis module pre-processes the data A through the data processing module, extracts the power A1, ripple voltage A2, sending time A3, IP address A4 and byte length A5, receiving time A6 of the data A, and extracts the corresponding power threshold A11, ripple voltage threshold A21 and byte length threshold A51, delay time threshold T in the database;

[0007] b. The abnormal screening module calculates the interference value A8 of the data. ,in represents the error calibration value of the interference value A8;

[0008] A8 greater than 0.1 indicates that the data is greatly disturbed, and data A is identified as invalid and filtered out. At the same time, the next step of data detection is stopped, and a warning signal is sent to the result evaluation module;

[0009] A8 less than 0.1 indicates that the data is less disturbed, and the next step of data detection is carried out;

[0010] c. The data verification module calculates the abnormal value A7 of the data. ,in Indicates the error correction value of the abnormal value A7. If A7 is less than 0.2, it means the data is normal. Otherwise, it means the data is abnormal. The data A is corrected by the data correction module, and the complete time series data is sent to the result evaluation module.

[0011] S3, the result evaluation module uses the complete time series data as the correct evaluation data to comprehensively evaluate the decision-making ability of the human-machine collaborative intelligent body.

[0012] Furthermore, the data correction module performs the following specific steps to correct the data A:

[0013] C1, retrieves the corresponding measurement data group in the database according to IP address A4, and further selects the measurement data group B with the same voltage through ripple voltage A2;

[0014] C2, calculate and extract the mean B1, median B2, mode B3 of the measurement data group B, and then calculate the predicted value B4, B4=B1×0.4+B2×0.3+B3×0.3;

[0015] C3, replaces data A with the predicted value B4, and inserts it into the time node position of the measured data corresponding to IP address A4 according to the sending time A3, forms the data of a complete time series, and sends it to the result evaluation module.

[0016] Furthermore, when the result evaluation module receives a warning signal, the abnormal location is displayed according to the IP address A4, and the data acquisition module is controlled to re-collect data A. When the result evaluation module receives the warning signal again, the above steps are repeated. Once the same IP address A4 receives the warning signal three times in a row, the evaluation is stopped and the cause of the abnormality is displayed, reminding relevant personnel to check and repair the network transmission device and environmental interference factors for the evaluation.

[0017] Furthermore, the system of the evaluation method includes a task setting module, a data acquisition module, a data analysis module, a result evaluation module and a database, wherein the data analysis module includes a data processing module, an abnormality screening module, a data verification module and a data correction module, and the signal transmission between modules adopts TCP / IP network protocol transmission.

[0018] Furthermore, the specific steps of calculating the formula weight coefficient of the interference value A8 of the data by the abnormal screening module are as follows;

[0019] b1, when other environmental factors remain unchanged, the ripple voltage and delay time of the collected data are set unchanged, and the power of the collected data is collected, with 10 data as a group;

[0020] Similarly, two sets of data of ripple voltage and delay time are collected respectively;

[0021] b2, unify the three sets of data into a scalar of the difference between the power, ripple voltage, and delay time of the data and the corresponding threshold, and calculate the ratio of the difference to the threshold respectively, and use the ratio of the power, ripple voltage, and delay time of the data as the scalar;

[0022] b3, taking the interference of the data during normal operation of the evaluation system for human-machine collaborative intelligent body in the same environment as the reference value, calculate the interference influence on the data when the power ratio of the data changes when the ripple voltage ratio and the delay time ratio are the same; similarly, calculate the influence of the ripple voltage ratio and the delay time ratio respectively;

[0023] A set of weight coefficients is initially set according to three different impact values;

[0024] b4, repeat steps b1-b3, and continuously iterate and adjust the weight coefficient until the adjustment of the weight coefficient has little effect on the interference of the data. Finally, determine the weight coefficient and calibration value, and obtain the interference value formula of the abnormal screening module to calculate the data, that is, ,in Indicates the error calibration value of the interference value A8.

[0025] Further, when the power A1 or the ripple voltage A2 of data A is normal, the difference of A1-A11 or A2-A21 in the formula of the interference value A8 is set to 0. Similarly, when the extension time of data A is normal, the value of A6-A3-T is set to 0.

[0026] The power threshold A11 and the ripple voltage threshold A21 are both range values. When the power A1 is lower than the minimum value of A11 or higher than the maximum value of A11, it is considered that the power A1 is abnormal. Similarly, when the ripple voltage A2 is lower than the minimum value of A21 or higher than the maximum value of A21, it is considered that the power A2 is abnormal.

[0027] Furthermore, the byte length threshold A51 is a range value. When the byte length A5 is lower than the minimum value of A51 or higher than the maximum value of A51, the byte length A5 is considered to be abnormal. At the same time, when the byte length A5 is normal, the data verification module calculates the value of A51-A5 in the abnormal value A7 of the data and sets it to 0.

[0028] Due to the adoption of the above technical solution, the present invention has the following advantages compared with the prior art:

[0029] 1. Through the analysis of the power A1, ripple voltage A2, sending time A3, IP address A4, byte length A5 and receiving time A6 of data A, the interference value formula of the data is used to timely obtain the interference degree of the data, and on this basis, the abnormal value of the data is further calculated, and the data that is severely interfered is timely re-collected, and the data A that suffers from data poisoning is promptly corrected through the data correction module, and the complete time series data is sent to the result evaluation module to ensure the consistency and accuracy of the data, thereby improving the accuracy and reliability of the evaluation results of the human-machine collaborative intelligent body. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 The present invention is a flow chart of an evaluation method for a human-machine collaborative intelligent agent.

[0031] Figure 2 This is a module diagram of an evaluation method for a human-machine collaborative intelligent agent according to the present invention.

[0032] Figure 3 This is a data analysis module diagram of a method for evaluating a human-machine collaborative intelligent body according to the present invention. DETAILED DESCRIPTION

[0033] The above and other technical contents, features and effects of the present invention are described in detail below with reference to the attached Figures 1 to 3 The detailed description of the embodiments will clearly show that the structural contents mentioned in the following embodiments are all based on the drawings in the specification.

[0034] In order to solve the problem of environmental interference or "data poisoning" in the evaluation of human-machine collaborative intelligent agents, it is necessary to first distinguish environmental interference and "data poisoning" in a targeted manner, and then solve them respectively. Therefore, S1 sets the human-machine collaborative intelligent agent decision-making task with preset environmental parameters and preset task difficulty through the task setting module. During the test, the data acquisition module collects data A of the human-machine collaborative intelligent agent in real time;

[0035] S2, a, the data analysis module pre-processes the data A through the data processing module, extracts the power A1, ripple voltage A2, sending time A3, IP address A4 and byte length A5, receiving time A6 of the data A, and extracts the corresponding power threshold A11, ripple voltage threshold A21 and byte length threshold A51, delay time threshold T in the database;

[0036] b. The abnormal screening module calculates the interference value A8 of the data. ,in represents the error calibration value of the interference value A8;

[0037] A8 greater than 0.1 indicates that the data is greatly disturbed, and data A is identified as invalid and filtered out. At the same time, the next step of data detection is stopped, and a warning signal is sent to the result evaluation module;

[0038] A8 less than 0.1 indicates that the data is less disturbed, and the next step of data detection is carried out;

[0039] c. The data verification module calculates the abnormal value A7 of the data. ,in Indicates the error correction value of the abnormal value A7. If A7 is less than 0.2, it means the data is normal. Otherwise, it means the data is abnormal. The data A is corrected by the data correction module, and the complete time series data is sent to the result evaluation module.

[0040] S3, the result evaluation module uses the complete time series data as the correct evaluation data to comprehensively evaluate the decision-making ability of the human-machine collaborative intelligent body.

[0041] Furthermore, the data correction module performs the following specific steps to correct the data A:

[0042] C1, retrieves the corresponding measurement data group in the database according to IP address A4, and further selects the measurement data group B with the same voltage through ripple voltage A2;

[0043] C2, calculate and extract the mean B1, median B2, mode B3 of the measurement data group B, and then calculate the predicted value B4, B4=B1×0.4+B2×0.3+B3×0.3;

[0044] C3, replaces data A with the predicted value B4, and inserts it into the time node position of the measured data corresponding to IP address A4 according to the sending time A3, forms the data of a complete time series, and sends it to the result evaluation module.

[0045] Furthermore, when the result evaluation module receives a warning signal, the abnormal location is displayed according to the IP address A4, and the data acquisition module is controlled to re-collect data A. When the result evaluation module receives the warning signal again, the above steps are repeated. Once the same IP address A4 receives the warning signal three times in a row, the evaluation is stopped and the cause of the abnormality is displayed, reminding relevant personnel to check and repair the network transmission device and environmental interference factors for the evaluation.

[0046] Furthermore, the system of the evaluation method includes a task setting module, a data acquisition module, a data analysis module, a result evaluation module and a database, wherein the data analysis module includes a data processing module, an abnormality screening module, a data verification module and a data correction module, and the signal transmission between modules adopts TCP / IP network protocol transmission.

[0047] Furthermore, the specific steps of calculating the formula weight coefficient of the interference value A8 of the data by the abnormal screening module are as follows;

[0048] b1, when other environmental factors remain unchanged, the ripple voltage and delay time of the collected data are set unchanged, and the power of the collected data is collected, with 10 data as a group;

[0049] Similarly, two sets of data of ripple voltage and delay time are collected respectively;

[0050] b2, unify the three sets of data into a scalar of the difference between the power, ripple voltage, and delay time of the data and the corresponding threshold, and calculate the ratio of the difference to the threshold respectively, and use the ratio of the power, ripple voltage, and delay time of the data as the scalar;

[0051] b3, taking the interference of the data during normal operation of the evaluation system for human-machine collaborative intelligent body in the same environment as the reference value, calculate the interference influence on the data when the power ratio of the data changes when the ripple voltage ratio and the delay time ratio are the same; similarly, calculate the influence of the ripple voltage ratio and the delay time ratio respectively;

[0052] A set of weight coefficients is initially set according to three different impact values;

[0053] b4, repeat steps b1-b3, and continuously iterate and adjust the weight coefficient until the adjustment of the weight coefficient has little effect on the interference of the data. Finally, determine the weight coefficient and calibration value, and obtain the interference value formula of the abnormal screening module to calculate the data, that is, ,in Indicates the error calibration value of the interference value A8.

[0054] Further, when the power A1 or the ripple voltage A2 of data A is normal, the difference of A1-A11 or A2-A21 in the formula of the interference value A8 is set to 0. Similarly, when the extension time of data A is normal, the value of A6-A3-T is set to 0.

[0055] The power threshold A11 and the ripple voltage threshold A21 are both range values. When the power A1 is lower than the minimum value of A11 or higher than the maximum value of A11, it is considered that the power A1 is abnormal. Similarly, when the ripple voltage A2 is lower than the minimum value of A21 or higher than the maximum value of A21, it is considered that the power A2 is abnormal.

[0056] Furthermore, the byte length threshold A51 is a range value. When the byte length A5 is lower than the minimum value of A51 or higher than the maximum value of A51, the byte length A5 is considered to be abnormal. At the same time, when the byte length A5 is normal, the data verification module calculates the value of A51-A5 in the abnormal value A7 of the data and sets it to 0.

[0057] When the present invention is used specifically, on the basis of the prior art, a human-machine collaborative intelligent body decision-making task with preset environmental parameters and preset task difficulty is set through a task setting module. During the test, the data collection module collects data A of the human-machine collaborative intelligent body in real time;

[0058] The data analysis module pre-processes the data A through the data processing module, extracts the power A1, ripple voltage A2, sending time A3, IP address A4, byte length A5, receiving time A6 of the data A, and extracts the corresponding power threshold A11, ripple voltage threshold A21, byte length threshold A51, and delay time threshold T in the database; the abnormal screening module calculates the interference value A8 of the data, ,in represents the error calibration value of the interference value A8;

[0059] A8 greater than 0.1 indicates that the data is greatly disturbed, and data A is identified as invalid and filtered out. At the same time, the next step of data detection is stopped, and a warning signal is sent to the result evaluation module;

[0060] A8 less than 0.1 indicates that the data is less disturbed, and the next step of data detection is carried out;

[0061] The data verification module calculates the abnormal value A7 of the data. ,in Indicates the error correction value of the abnormal value A7. If A7 is less than 0.2, it means the data is normal. Otherwise, it means the data is abnormal. The data A is corrected by the data correction module, and the complete time series data is sent to the result evaluation module.

[0062] The result evaluation module uses the complete time series data as the correct evaluation data, conducts a comprehensive evaluation of the decision-making ability of the human-machine collaborative intelligent body, uses the data interference value formula to timely obtain the degree of data interference, and further calculates the data outliers on this basis, and promptly re-collects the data that is severely interfered with, and promptly corrects the data A through the data correction module if it suffers from data poisoning, and sends the complete time series data to the result evaluation module to ensure the consistency and accuracy of the data, thereby improving the accuracy and reliability of the evaluation results of the human-machine collaborative intelligent body.

[0063] The above is a further detailed description of the present invention in combination with a specific implementation method, and it cannot be determined that the specific implementation of the present invention is limited to this; for technical personnel in the technical field to which the present invention belongs and related technical fields, based on the technical solution of the present invention, the expansion and replacement of operating methods and data should all fall within the protection scope of the present invention.

Claims

1. A method for evaluating a human-machine collaborative agent, characterized in that: The specific steps of this method are as follows: S1, setting the human-machine collaborative intelligent agent decision-making task with preset environmental parameters and preset task difficulty through the task setting module. During the test, the data acquisition module collects data A of the human-machine collaborative intelligent agent in real time; S2, a, the data analysis module pre-processes the data A through the data processing module, extracts the power A1, ripple voltage A2, sending time A3, IP address A4 and byte length A5, receiving time A6 of the data A, and extracts the corresponding power threshold A11, ripple voltage threshold A21 and byte length threshold A51, delay time threshold T in the database; b. The abnormal screening module calculates the interference value A8 of the data. ,in represents the error calibration value of the interference value A8; A8 greater than 0.1 indicates that the data is greatly disturbed, and data A is identified as invalid and filtered out. At the same time, the next step of data detection is stopped, and a warning signal is sent to the result evaluation module; A8 less than 0.1 indicates that the data is less disturbed, and the next step of data detection is carried out; c. The data verification module calculates the abnormal value A7 of the data. ,in Indicates the error correction value of the abnormal value A7. If A7 is less than 0.2, it means the data is normal. Otherwise, it means the data is abnormal. The data A is corrected by the data correction module, and the complete time series data is sent to the result evaluation module. S3, the result evaluation module uses the complete time series data as the correct evaluation data to comprehensively evaluate the decision-making ability of the human-machine collaborative intelligent body.

2. A method for evaluating a human-machine collaborative agent as claimed in claim 1, characterized in that: The specific steps of the data correction module correcting data A are as follows: C1, retrieves the corresponding measurement data group in the database according to IP address A4, and further selects the measurement data group B with the same voltage through ripple voltage A2; C2, calculate and extract the mean B1, median B2, mode B3 of the measurement data group B, and then calculate the predicted value B4, B4=B1×0.4+B2×0.3+B3×0.3; C3, replaces data A with the predicted value B4, and inserts it into the time node position of the measured data corresponding to IP address A4 according to the sending time A3, forms the data of a complete time series, and sends it to the result evaluation module.

3. The evaluation method for a human-machine collaborative agent as claimed in claim 1, characterized in that: When the result evaluation module receives the warning signal, it displays the abnormal location according to the IP address A4 and controls the data acquisition module to re-collect data A. When the result evaluation module receives the warning signal again, it repeats the above steps. Once the same IP address A4 receives the warning signal three times in a row, the evaluation is stopped and the cause of the abnormality is displayed to remind relevant personnel to check and repair the network transmission device and environmental interference factors of the evaluation.

4. The evaluation method for a human-machine collaborative agent as claimed in claim 1, characterized in that: The system of the evaluation method includes a task setting module, a data acquisition module, a data analysis module, a result evaluation module and a database, wherein the data analysis module includes a data processing module, an abnormal screening module, a data verification module and a data correction module, and the signal transmission between modules adopts TCP / IP network protocol transmission.

5. The evaluation method for a human-machine collaborative agent as claimed in claim 1, characterized in that: The specific steps of calculating the formula weight coefficient of the interference value A8 of the data by the abnormal screening module are as follows; b1, when other environmental factors remain unchanged, the ripple voltage and delay time of the collected data are set unchanged, and the power of the collected data is collected, with 10 data as a group; Similarly, two sets of data of ripple voltage and delay time are collected respectively; b2, unify the three sets of data into a scalar of the difference between the power, ripple voltage, and delay time of the data and the corresponding threshold, and calculate the ratio of the difference to the threshold respectively, and use the ratio of the power, ripple voltage, and delay time of the data as the scalar; b3, taking the interference of the data during normal operation of the evaluation system for human-machine collaborative intelligent body in the same environment as the reference value, calculate the interference influence on the data when the power ratio of the data changes when the ripple voltage ratio and the delay time ratio are the same; similarly, calculate the influence of the ripple voltage ratio and the delay time ratio respectively; A set of weight coefficients is initially set according to three different impact values; b4, repeat steps b1-b3, and continuously iterate and adjust the weight coefficient until the adjustment of the weight coefficient has little effect on the interference of the data. Finally, determine the weight coefficient and calibration value, and obtain the interference value formula of the abnormal screening module to calculate the data, that is, ,in Indicates the error calibration value of the interference value A8.

6. The evaluation method for a human-machine collaborative agent as claimed in claim 1, characterized in that: When the power A1 or ripple voltage A2 of data A is normal, the difference of A1-A11 or A2-A21 in the formula of interference value A8 is set to 0. Similarly, when the extension time of data A is normal, the value of A6-A3-T is set to 0. The power threshold A11 and the ripple voltage threshold A21 are both range values. When the power A1 is lower than the minimum value of A11 or higher than the maximum value of A11, it is considered that the power A1 is abnormal. Similarly, when the ripple voltage A2 is lower than the minimum value of A21 or higher than the maximum value of A21, it is considered that the power A2 is abnormal.

7. The evaluation method for a human-machine collaborative agent as claimed in claim 1, characterized in that: The byte length threshold A51 is a range value. When the byte length A5 is lower than the minimum value of A51 or higher than the maximum value of A51, it is considered that the byte length A5 is abnormal. At the same time, when the byte length A5 is normal, the data verification module calculates the value of A51-A5 in the abnormal value A7 of the data and sets it to 0.

Images