Production command agent test and evaluation method, device, equipment and medium

Abstract

The invention discloses a production command agent test and evaluation method, device and equipment and a medium, and belongs to the technical field of agents, and the method comprises the steps: combining any evaluation dimension in an evaluation dimension set with any test scene type in a test scene type set, and obtaining an evaluation factor combination set; performing parameter matching in an agent configuration parameter pool according to the evaluation dimension and the test scene type corresponding to the evaluation factor combination, and generating a plurality of agent configuration parameter groups corresponding to the evaluation factor combination; the production command agents corresponding to the agent configuration parameter sets are evaluated and tested, and evaluation index values corresponding to the agent configuration parameter sets are obtained; and according to the evaluation factor combination set, the agent configuration parameter group and the evaluation index value, generating total evaluation test information, and according to the total test information, testing and evaluating the to-be-evaluated production command agent. Therefore, by implementing the method, the evaluation reliability of the production command agent can be improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent agent technology, and in particular to a method, apparatus, equipment and medium for testing and evaluating intelligent agents for production command. Background Technology

[0002] Existing technologies have some shortcomings in the evaluation and testing of production command agents. For example, traditional evaluation methods are relatively simplistic, often focusing only on a few fixed evaluation dimensions, making it difficult to comprehensively consider the agent's performance in complex and ever-changing real-world production scenarios. Moreover, the coverage of test scenario types is not broad enough, failing to simulate various extreme situations and unexpected events that may occur during production.

[0003] Furthermore, existing methods for generating agent configuration parameters lack flexibility and diversity, failing to dynamically generate the most suitable parameter sets based on different combinations of evaluation factors. This results in evaluation test results that do not accurately reflect the agent's true performance. Moreover, the determination of evaluation index values ​​may suffer from imprecise quantitative analysis, impacting the reliability of the evaluation results.

[0004] Furthermore, when faced with new production command agents or when optimizing and upgrading existing agents, existing evaluation and testing methods and systems are insufficient for rapid and effective evaluation and adjustment, failing to provide timely and accurate guidance for optimizing agent configuration parameters. All these issues hinder the efficient application and performance improvement of production command agents in actual production. Summary of the Invention

[0005] This invention provides a method, apparatus, equipment, and medium for testing and evaluating production command intelligence agents, which can broadly cover actual production conditions and improve the reliability of production command intelligence agent evaluation.

[0006] This invention provides a method for testing and evaluating a production command intelligent agent, including: Each evaluation dimension in the evaluation dimension set is combined with any test scenario type in the test scenario type set to obtain an evaluation factor combination set; wherein, the evaluation factor combination set contains at least one evaluation factor combination. Based on the evaluation dimension and test scenario type corresponding to each combination of evaluation factors, parameter matching is performed in the pre-built agent configuration parameter pool to generate multiple agent configuration parameter groups corresponding to each combination of evaluation factors. The production command intelligent agents corresponding to each of the intelligent agent configuration parameter groups are evaluated and tested to obtain the evaluation index values ​​corresponding to each of the intelligent agent configuration parameter groups. Based on the set of evaluation factors, the configuration parameter groups of each agent, and the values ​​of each evaluation index, a total evaluation test information is generated, and the production command agent to be evaluated is tested and evaluated based on the total evaluation test information.

[0007] This invention, through combining any evaluation dimension from the evaluation dimension set with any test scenario type from the test scenario type set, constructs a comprehensive set of evaluation factor combinations covering different evaluation perspectives and test environments, laying the foundation for subsequent systematic testing. By matching parameters in a pre-built agent configuration parameter pool according to the evaluation dimension and test scenario type corresponding to each evaluation factor combination, diverse agent configurations can be dynamically generated for different evaluation factor combinations, improving the relevance and coverage of the tests. By conducting separate evaluation tests on the production command agents corresponding to each agent configuration parameter group, specific performance data of agents under different configurations in various scenarios can be obtained, achieving refined performance measurement. By generating total evaluation test information based on the evaluation factor combination set, each agent configuration parameter group, and each evaluation index value, and testing and evaluating the production command agent to be evaluated based on this total evaluation test information, all test data can be integrated to form a comprehensive evaluation benchmark, which is then used to evaluate the performance of the agent to be evaluated. Compared to existing technologies that suffer from limited coverage and lack of flexibility in parameter generation, this application can broadly cover actual production conditions, improving the reliability of production command agent evaluation.

[0008] Furthermore, before combining any evaluation dimension from the evaluation dimension set with any test scenario type from the test scenario type set to obtain the evaluation factor combination set, the method further includes: Obtain an evaluation dimension set and a test scenario type set; wherein the evaluation dimension set contains at least one evaluation dimension, and the test scenario type set contains at least one test scenario type.

[0009] By obtaining the set of evaluation dimensions and the set of test scenario types, the embodiments of the present invention can clarify the input range and preconditions of the test, and ensure the integrity and executability of subsequent combinations and testing processes.

[0010] Further, the step of matching parameters in a pre-built agent configuration parameter pool based on the evaluation dimension and test scenario type corresponding to each evaluation factor combination to generate multiple agent configuration parameter groups corresponding to each evaluation factor combination includes: Select at least one agent configuration parameter type from the agent configuration parameter pool that matches both the evaluation dimension and the test scenario type corresponding to the target evaluation factor combination; wherein, the target evaluation factor combination is any evaluation factor combination in the evaluation factor combination set; Obtain the parameter value generation rules corresponding to each of the intelligent agent configuration parameter types, and generate multiple parameter values ​​corresponding to each of the intelligent agent configuration parameter types according to the parameter value generation rules; The parameter values ​​are arranged and combined according to the preset permutation and combination rules to generate multiple agent configuration parameter groups corresponding to the target evaluation factor combination.

[0011] This invention, through matching parameter types from a parameter pool, generating parameter values, and arranging and combining them to generate parameter groups, enables a systematic generation process from parameter type to parameter value and then to parameter group, thereby improving the flexibility of parameter configuration and the comprehensiveness of testing.

[0012] Furthermore, after generating the total evaluation test information based on the set of evaluation factor combinations, the configuration parameter groups of each agent, and the values ​​of each evaluation index, the method further includes: Obtain the first agent configuration parameter group, first evaluation dimension, and first test scenario type corresponding to the verification production command agent; Based on the first intelligent agent configuration parameter group, the first evaluation dimension and the first test scenario type, the first evaluation index value corresponding to the verification production command intelligent agent is matched in the total evaluation test information; The verification production command intelligence agent is evaluated and tested to obtain the second evaluation index value; The first evaluation index value and the second evaluation index value are compared. If the comparison result does not meet the preset conditions, the compensation configuration parameter corresponding to the first evaluation dimension is added to the agent configuration parameter pool.

[0013] This invention, through the introduction of a verification production command agent after generating total evaluation test information to compare actual and predicted values, and updating the parameter pool when conditions are not met, enables the self-verification and continuous optimization of the evaluation system, thereby improving the accuracy and reliability of the total evaluation test information.

[0014] Further, the evaluation and testing of the production command agents corresponding to each of the agent configuration parameter groups to obtain the evaluation index values ​​corresponding to each agent configuration parameter group includes: Input multimodal test data matching the target test scenario type into the target production command agent; wherein, the target production command agent is the production command agent corresponding to the second agent configuration parameter group, the second agent configuration parameter group is any agent configuration parameter group in each of the agent configuration parameter groups, and the target test scenario type is any test scenario type in the test scenario type set; Receive the decision response results output by the target production command intelligence agent; The decision response results are quantitatively analyzed according to preset evaluation rules to obtain the evaluation index values ​​corresponding to the second agent configuration parameter group.

[0015] The embodiments of the present invention input multimodal test data into the target intelligent agent and perform quantitative analysis on its decision response results to obtain evaluation index values, thereby enabling objective and accurate quantitative evaluation of the intelligent agent's performance.

[0016] Further, the step of generating total evaluation test information based on the set of evaluation factor combinations, the configuration parameter groups of each agent, and the values ​​of each evaluation index includes: Based on the configuration parameter groups of each agent corresponding to the target evaluation factor combination, and the evaluation index values ​​corresponding to the configuration parameter groups of each agent corresponding to the target evaluation factor combination, sub-evaluation test information corresponding to the target evaluation factor combination is generated. The sub-evaluation test information corresponding to each of the aforementioned evaluation factors is aggregated to generate total evaluation test information.

[0017] This invention generates sub-evaluation test information by first combining target evaluation factors, and then aggregating all sub-information to generate total evaluation test information. This generates a well-structured total evaluation test information, providing an evaluation benchmark for subsequent agent testing.

[0018] Furthermore, the testing and evaluation of the performance of the production command agent to be evaluated based on the total evaluation test information includes: Obtain the configuration parameter group of the third agent, the second evaluation dimension, the second test scenario type, and the maximum evaluation threshold corresponding to the production command agent to be evaluated; Based on the third agent configuration parameter group, the second evaluation dimension, and the second test scenario type, the third evaluation index value corresponding to the production command agent to be evaluated is matched in the total evaluation test information; The third evaluation index value is compared with the maximum evaluation threshold to generate agent evaluation information for the production command agent to be evaluated; wherein, the agent evaluation information is used to indicate whether the configuration parameters of the production command agent to be evaluated should be adjusted.

[0019] This invention, through obtaining the configuration and scenario of the agent to be evaluated, matches index values ​​in the total evaluation test information and compares them with thresholds to generate evaluation information. This enables rapid performance evaluation and qualification judgment of new agents, and provides guidance for parameter adjustment.

[0020] Another embodiment of the present invention provides a production command intelligent agent testing and evaluation device, including: an evaluation factor combination module, a configuration parameter group module, an evaluation index value module, and an evaluation test module; The evaluation factor combination module is used to combine any evaluation dimension in the evaluation dimension set with any test scenario type in the test scenario type set to obtain an evaluation factor combination set; wherein, the evaluation factor combination set contains at least one evaluation factor combination. The configuration parameter group module is used to perform parameter matching in a pre-built agent configuration parameter pool according to the evaluation dimension and test scenario type corresponding to each evaluation factor combination, and generate multiple agent configuration parameter groups corresponding to each evaluation factor combination. The evaluation index value module is used to evaluate and test the production command intelligent agent corresponding to each intelligent agent configuration parameter group to obtain the evaluation index value corresponding to each intelligent agent configuration parameter group. The evaluation and testing module is used to generate total evaluation and testing information based on the set of evaluation factors, the configuration parameter groups of each agent, and the evaluation index values, and to test and evaluate the production command agent to be evaluated based on the total evaluation and testing information.

[0021] Another embodiment of the present invention provides a terminal device, including: a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the steps of a production command intelligent agent testing and evaluation method as described in the present invention.

[0022] Another embodiment of the present invention provides a computer-readable storage medium item, including: a stored computer program, which, when the computer program is running, controls the device where the computer-readable storage medium is located to perform the steps of a production command intelligent agent testing and evaluation method of the present invention. Attached Figure Description

[0023] Figure 1 A flowchart illustrating an embodiment of the production command intelligent agent testing and evaluation method provided by the present invention; Figure 2 This is a schematic diagram of one embodiment of the production command intelligent agent testing and evaluation device provided by the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0026] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0027] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0028] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0029] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), similarly, "multiple sets" refers to two or more (including two sets), and "multiple pieces" refers to two or more (including two pieces).

[0030] See Figure 1 To address the problems of insufficient coverage and lack of flexibility in parameter generation in existing technologies, an embodiment of the present invention provides a method for testing and evaluating production command intelligent agents, including steps S101 to S104: Step S101: Combine any evaluation dimension in the evaluation dimension set with any test scenario type in the test scenario type set to obtain an evaluation factor combination set; wherein the evaluation factor combination set contains at least one evaluation factor combination.

[0031] Preferably, before combining any evaluation dimension from the evaluation dimension set with any test scenario type from the test scenario type set to obtain the evaluation factor combination set, the method further includes: Obtain an evaluation dimension set and a test scenario type set; wherein the evaluation dimension set contains at least one evaluation dimension, and the test scenario type set contains at least one test scenario type.

[0032] In one optional implementation, the evaluation dimensions can encompass multiple aspects, including the decision-making accuracy, response speed, resource utilization, and interpretability of the production command agent. Decision-making accuracy measures the production command agent's ability to make correct decisions when faced with various production command tasks, which directly relates to the smooth operation of the production process and the quality of the final product. Response speed reflects how quickly the production command agent can provide a decision response after receiving task instructions; in production scenarios with high time requirements, a rapid response speed is crucial. Resource utilization reflects the efficiency of the production command agent in utilizing production resources such as human, material, and financial resources; efficient resource utilization can reduce production costs and improve the economic benefits of the enterprise. Interpretability refers to the ability of the production command agent's decision-making process and rationale to be understood by humans, which is particularly important in production fields with high requirements for security and reliability.

[0033] In one optional implementation, test scenario types may include normal production scenarios, sudden failure scenarios, emergency order scenarios, and resource shortage scenarios. Normal production scenarios represent the most common production conditions and primarily test the performance of the production command agent during daily production. Sudden failure scenarios simulate situations where production equipment suddenly malfunctions, examining the production command agent's ability to quickly make response decisions and minimize the impact of the failure on production. Emergency order scenarios require the production command agent to adjust production plans within a short period to prioritize the needs of urgent orders, testing its flexibility and responsiveness. Resource shortage scenarios simulate situations where certain critical resources are insufficient during production, verifying the production command agent's production scheduling capabilities under resource-constrained conditions.

[0034] It's important to note that by acquiring these different evaluation dimensions and test scenario types and combining them in pairs, a rich variety of evaluation factor combinations can be obtained. These combinations can more comprehensively reflect the performance of the production command agent under various actual production conditions, providing a solid foundation for subsequent agent configuration parameter groups and evaluation testing. For example, combining the decision-making accuracy evaluation dimension with a sudden failure scenario can specifically examine the production command agent's ability to make correct decisions when production equipment suddenly malfunctions. In this case, the production command agent must not only respond quickly but also accurately determine the type and severity of the failure, thereby formulating a reasonable response strategy. As another example, combining the response speed evaluation dimension with an emergency order scenario focuses on testing whether the production command agent can quickly adjust the production plan and provide a rapid decision-making response when faced with an emergency order.

[0035] It should be noted that when determining the evaluation dimensions and test scenario types, the specific content can be determined by surveying experts in the production field, analyzing historical production data, or referring to relevant industry standards. Furthermore, the evaluation dimensions and test scenario types can be appropriately adjusted and optimized based on the characteristics of different production enterprises and processes to ensure that the evaluation and testing results accurately reflect the actual performance of the intelligent agent.

[0036] Step S102: Based on the evaluation dimension and test scenario type corresponding to each evaluation factor combination, perform parameter matching in the pre-built agent configuration parameter pool to generate multiple agent configuration parameter groups corresponding to each evaluation factor combination.

[0037] Preferably, the step of matching parameters in a pre-built agent configuration parameter pool according to the evaluation dimension and test scenario type corresponding to each evaluation factor combination to generate multiple agent configuration parameter groups corresponding to each evaluation factor combination includes: Select at least one agent configuration parameter type from the agent configuration parameter pool that matches both the evaluation dimension and the test scenario type corresponding to the target evaluation factor combination; wherein, the target evaluation factor combination is any evaluation factor combination in the evaluation factor combination set; Obtain the parameter value generation rules corresponding to each of the intelligent agent configuration parameter types, and generate multiple parameter values ​​corresponding to each of the intelligent agent configuration parameter types according to the parameter value generation rules; The parameter values ​​are arranged and combined according to the preset permutation and combination rules to generate multiple agent configuration parameter groups corresponding to the target evaluation factor combination.

[0038] It's important to note that the pre-built agent configuration parameter pool is a collection of various possible agent configuration parameter types. These parameters cover all aspects of the production command agent, such as algorithm parameters, data processing methods, and decision rules. When generating agent configuration parameter sets, the pool is used to select matching agent configuration parameter types based on the evaluation dimensions and test scenario types in each evaluation factor combination. For example, for the evaluation factor combination of decision accuracy assessment and sudden failure scenarios, parameter types that improve the production command agent's ability to make accurate decisions in failure situations are selected from the agent configuration parameter pool, such as more refined fault diagnosis algorithm parameters and more reasonable decision rule weights. Conversely, for the evaluation factor combination of response speed assessment and emergency order scenarios, parameter types that accelerate the production command agent's response speed are selected, such as optimized data processing parameters and efficient task scheduling algorithms.

[0039] It should be noted that selecting at least one agent configuration parameter type from the agent configuration parameter pool that matches both the evaluation dimension and test scenario type corresponding to the target evaluation factor combination means selecting specific evaluation dimensions and test scenario types as targets during the evaluation process, thereby clarifying the basis for selection from the agent configuration parameter pool. For example, if the evaluation dimension and test scenario type corresponding to the evaluation factor combination are the system's response speed and a high-concurrency environment, respectively, then the agent configuration parameter type selected from the agent configuration parameter pool that matches both the response speed and the high-concurrency environment, such as the number of threads and cache size, will be chosen.

[0040] It should be noted that obtaining the parameter value generation rules corresponding to each of the aforementioned agent configuration parameter types, and generating multiple parameter values ​​corresponding to each of the aforementioned agent configuration parameter types according to the aforementioned parameter value generation rules, refers to determining the specific method or logic for generating parameter values ​​for each selected agent configuration parameter type. For example, for the agent configuration parameter type of thread count, its parameter value generation rule can be based on a multiple relationship of the number of system cores, such as setting it to a random value between 1 and 4 times the number of cores.

[0041] It should be noted that, according to the preset permutation and combination rules, generating multiple agent configuration parameter groups corresponding to the target evaluation factor combinations means combining all parameter values ​​into multiple complete agent configuration parameters according to a predetermined permutation and combination method to meet diverse testing needs. For example, if the parameter values ​​for the number of threads are 2, 4, and 8, and the parameter values ​​for the cache size are 10MB and 20MB, then according to the permutation and combination rules, multiple agent configuration parameter groups can be generated, such as combinations with 2 threads and a cache size of 10MB, 4 threads and a cache size of 20MB, etc.

[0042] Step S103: Evaluate and test each production command agent corresponding to each of the agent configuration parameter groups to obtain the evaluation index value corresponding to each agent configuration parameter group.

[0043] Preferably, the step of evaluating and testing the production command agents corresponding to each of the agent configuration parameter groups to obtain evaluation index values ​​corresponding to each agent configuration parameter group includes: Input multimodal test data matching the target test scenario type into the target production command agent; wherein, the target production command agent is the production command agent corresponding to the second agent configuration parameter group, the second agent configuration parameter group is any agent configuration parameter group in each of the agent configuration parameter groups, and the target test scenario type is any test scenario type in the test scenario type set; Receive the decision response results output by the target production command intelligence agent; The decision response results are quantitatively analyzed according to preset evaluation rules to obtain the evaluation index values ​​corresponding to the second agent configuration parameter group.

[0044] It should be noted that after generating multiple agent configuration parameter groups, actual evaluation and testing are required for the production command agent corresponding to each agent configuration parameter group. The evaluation and testing process involves simulating various real-world production scenarios, allowing the production command agent to operate in these scenarios, and collecting relevant data according to the previously determined evaluation dimensions to obtain the evaluation index values ​​corresponding to each agent configuration parameter group.

[0045] In one alternative implementation, various methods can be used for evaluation testing. For example, a virtual production environment can be built that highly simulates actual production processes and various possible scenarios. The production command agent corresponding to each agent's configuration parameter group is placed in this virtual production environment and run, with corresponding tasks and conditions set according to different test scenario types. In normal production scenarios, the decision-making accuracy, response speed, and resource utilization of the production command agent in daily production processes are observed. In sudden failure scenarios, equipment failures are simulated, and the time and effectiveness of the production command agent in detecting the failure, analyzing its causes, and developing solutions are recorded to evaluate its ability to handle emergencies. In urgent order scenarios, the performance of the production command agent in adjusting production plans, coordinating resource allocation, and ensuring timely order completion is examined. In resource shortage scenarios, how the production command agent optimizes resource use and adjusts production rhythm to ensure production continuity is examined.

[0046] It should be noted that inputting multimodal test data matching the target test scenario type into the target production command agent refers to inputting diverse data that meets the requirements of a specific test scenario into a designated production command agent. This data includes, but is not limited to, various forms such as images, text, voice, or sensor data, and can simulate complex situations in a real production environment. For example, when the target test scenario type is anomaly detection on a factory production line, multimodal data including equipment operating status data, real-time monitoring video, and operator voice commands can be input to verify the performance of the target production command agent in this scenario.

[0047] It should be noted that receiving the decision response results output by the target production command intelligence agent refers to obtaining the feedback information generated by the target production command intelligence agent based on the input multimodal test data. This feedback information may include specific action instructions, analysis conclusions, or optimization suggestions. For example, after inputting test data on production line scheduling, the target production command intelligence agent may output instructions to adjust the operating parameters of a certain piece of equipment, or propose suggestions to reallocate task priorities.

[0048] It should be noted that quantitative analysis of the decision response results according to preset evaluation rules refers to scoring or calculating indicators on the decision response results output by the target production command intelligence agent based on predefined evaluation standards and calculation methods, thereby obtaining quantitative evaluation results. For example, by comparing the degree of deviation between the decision response results of the target production command intelligence agent and the actual optimal solution, and combining dimensions such as response time and resource utilization rate, the final target evaluation indicator value can be calculated according to a preset weight formula.

[0049] Step S104: Generate total evaluation test information based on the set of evaluation factors, the configuration parameter groups of each agent, and the values ​​of each evaluation index, and test and evaluate the production command agent to be evaluated based on the total evaluation test information.

[0050] Preferably, after generating the total evaluation test information based on the set of evaluation factor combinations, the configuration parameter groups of each agent, and the values ​​of each evaluation index, the method further includes: Obtain the first agent configuration parameter group, first evaluation dimension, and first test scenario type corresponding to the verification production command agent; Based on the first intelligent agent configuration parameter group, the first evaluation dimension and the first test scenario type, the first evaluation index value corresponding to the verification production command intelligent agent is matched in the total evaluation test information; The verification production command intelligence agent is evaluated and tested to obtain the second evaluation index value; The first evaluation index value and the second evaluation index value are compared. If the comparison result does not meet the preset conditions, the compensation configuration parameter corresponding to the first evaluation dimension is added to the agent configuration parameter pool.

[0051] It should be noted that obtaining the first agent configuration parameter set, the first evaluation dimension, and the first test scenario type corresponding to the verification production command agent refers to extracting specific data related to the verification production command agent from the existing information set. This data includes parameter sets used to define the behavior of the production command agent, the specific direction or angle of evaluation, and the specific environment type in which the test takes place. For example, during execution, the system will select a set of parameters suitable for verifying the production command agent from multiple stored agent configuration parameter sets, while specifying its evaluation dimension as performance or resource consumption, and determining the test scenario type as a high-load or low-load environment.

[0052] It should be noted that, based on the first agent configuration parameter group, the first evaluation dimension, and the first test scenario type, the first evaluation index value matched with the verification production command agent in the total evaluation test information refers to finding the specific evaluation result matching the verification production command agent in the total evaluation test information based on the relevant data extracted above. For example, after the system inputs the agent configuration parameter group, selects the evaluation dimension, and the test scenario type, it automatically retrieves the corresponding evaluation index value from the total evaluation test information, such as the response time or task completion rate of an agent under a high-load scenario.

[0053] It should be noted that evaluating the verification production command agent to obtain the second evaluation index value means running the verification production command agent in an independent test environment and recording its actual performance data to generate a new evaluation index value. For example, the system deploys the verification production command agent in a simulated test environment, collects its performance data during operation, and calculates the corresponding evaluation index value, such as task success rate or resource utilization rate.

[0054] It should be noted that comparing the first and second evaluation metric values, and adding the compensation configuration parameter corresponding to the first evaluation dimension to the agent configuration parameter pool when the comparison result does not meet the preset conditions, means comparing the difference between the two evaluation metric values ​​to determine whether the existing overall evaluation test information needs to be adjusted or supplemented. When the existing overall evaluation test information needs to be adjusted or supplemented, the newly generated compensation parameter is added to the agent configuration parameter pool, and the evaluation process is rerun to improve the overall evaluation test information. For example, if there is a significant difference between the first and second evaluation metric values, the system will determine that the current overall evaluation test information is not accurate enough and trigger an update process to optimize the evaluation results. Adding the compensation parameter related to the performance dimension to the agent configuration parameter pool and re-executing the evaluation test ultimately generates more accurate overall evaluation test information for subsequent use.

[0055] Preferably, generating total evaluation test information based on the set of evaluation factor combinations, the configuration parameter groups of each agent, and the values ​​of each evaluation index includes: Based on the configuration parameter groups of each agent corresponding to the target evaluation factor combination, and the evaluation index values ​​corresponding to the configuration parameter groups of each agent corresponding to the target evaluation factor combination, sub-evaluation test information corresponding to the target evaluation factor combination is generated. The sub-evaluation test information corresponding to each of the aforementioned evaluation factors is aggregated to generate total evaluation test information.

[0056] It should be noted that generating sub-evaluation test information corresponding to the target evaluation factor combination refers to generating detailed test data that reflects the characteristics of a specific selected evaluation factor combination, combined with its corresponding multiple agent configuration parameter groups and their evaluation index values, providing a foundation for subsequent overall evaluation. For example, if the target evaluation factor combination involves high-precision sensor configuration and low-latency communication conditions, sub-evaluation test information related to these conditions can be generated, such as recording specific data like the time consumption, resource utilization, and task success rate of the production command agent in completing a specific task under this configuration.

[0057] It should be noted that aggregating the sub-evaluation test information corresponding to each combination of evaluation factors to generate total evaluation test information means integrating the sub-evaluation test information generated by each combination of evaluation factors to form a complete set of evaluation results, thereby comprehensively reflecting the overall performance of all combinations of evaluation factors. For example, sub-evaluation test information under various sensor configurations, different computing resource allocations, and various task priority settings can be summarized to obtain total evaluation test information covering all test scenarios, so as to further analyze the overall performance of the system and the direction of optimization.

[0058] Preferably, the step of testing and evaluating the performance of the production command intelligence agent to be evaluated based on the total evaluation test information includes: Obtain the configuration parameter group of the third agent, the second evaluation dimension, the second test scenario type, and the maximum evaluation threshold corresponding to the production command agent to be evaluated; Based on the third agent configuration parameter group, the second evaluation dimension, and the second test scenario type, the third evaluation index value corresponding to the production command agent to be evaluated is matched in the total evaluation test information; The third evaluation index value is compared with the maximum evaluation threshold to generate agent evaluation information for the production command agent to be evaluated; wherein, the agent evaluation information is used to indicate whether the configuration parameters of the production command agent to be evaluated should be adjusted.

[0059] It should be noted that obtaining the third-party configuration parameter set corresponding to the production command agent to be evaluated refers to extracting a set of specific configuration parameters related to the production command agent currently being evaluated from the system. These parameters may include operating rules, decision-making logic, or resource allocation strategies. For example, the obtained third-party configuration parameter set may contain specific values ​​for the scheduling priority settings, task allocation weights, and exception handling mechanisms of the production command agent in a specific scenario.

[0060] It should be noted that the second evaluation dimension refers to a specific type of indicator used to measure the performance of the production command agent being evaluated, such as efficiency, stability, or resource utilization. For example, the second evaluation dimension could be the degree to which the production command agent optimizes the completion time of production tasks under high load conditions, or its ability to balance the use of system resources across multiple scenario types.

[0061] It should be noted that the second test scenario type refers to a specific set of environments or conditions designed for the production command intelligence agent to be evaluated, used to simulate different situations in actual production, thereby comprehensively evaluating its performance. For example, the second test scenario type may include emergency response scenarios during sudden order peaks, fault-tolerant handling scenarios when some equipment on the production line fails, and resource scheduling scenarios when multiple tasks are concurrent.

[0062] It should be noted that the maximum evaluation threshold refers to a performance upper limit standard set by the system for the production command agent to be evaluated, used to determine whether it meets the expected requirements. For example, the maximum evaluation threshold can be a standard value for the maximum allowable task delay time or the minimum task completion rate of the production command agent in a specific scenario.

[0063] It should be noted that the third evaluation metric value refers to a specific numerical value calculated based on the third agent configuration parameter group, the second evaluation dimension, and the second test scenario type. It is used to quantify the performance of the production command agent to be evaluated. For example, the target evaluation metric value might be the percentage reduction in task completion time for the production command agent under simulated high-load scenarios, or the percentage increase in resource utilization achieved under resource-constrained conditions.

[0064] It should be noted that agent evaluation information refers to comprehensive data or reports generated based on the comparison results of the third evaluation index value and the maximum evaluation threshold, used to guide the subsequent adjustment of configuration parameters of the production command agent to be evaluated. For example, agent evaluation information can display the comprehensive performance of the current production command agent in various test scenarios and explicitly recommend adjusting its task allocation weights or adding relevant parameters for fault tolerance mechanisms.

[0065] This invention, through combining any evaluation dimension from the evaluation dimension set with any test scenario type from the test scenario type set, constructs a comprehensive set of evaluation factor combinations covering different evaluation perspectives and test environments, laying the foundation for subsequent systematic testing. By matching parameters in a pre-built agent configuration parameter pool according to the evaluation dimension and test scenario type corresponding to each evaluation factor combination, diverse agent configurations can be dynamically generated for different evaluation factor combinations, improving the relevance and coverage of the tests. By conducting separate evaluation tests on the production command agents corresponding to each agent configuration parameter group, specific performance data of agents under different configurations in various scenarios can be obtained, achieving refined performance measurement. By generating total evaluation test information based on the evaluation factor combination set, each agent configuration parameter group, and each evaluation index value, and testing and evaluating the production command agent to be evaluated based on this total evaluation test information, all test data can be integrated to form a comprehensive evaluation benchmark, which can then be used to evaluate the performance of the agent to be evaluated.

[0066] Optionally, in this embodiment of the invention, before combining any evaluation dimension in the evaluation dimension set with any test scenario type in the test scenario type set to obtain the evaluation factor combination set, the method further includes: Obtain an evaluation dimension set and a test scenario type set; wherein the evaluation dimension set contains at least one evaluation dimension, and the test scenario type set contains at least one test scenario type.

[0067] By obtaining the set of evaluation dimensions and the set of test scenario types, the embodiments of the present invention can clarify the input range and preconditions of the test, and ensure the integrity and executability of subsequent combinations and testing processes.

[0068] Optionally, in this embodiment of the invention, the step of matching parameters in a pre-built agent configuration parameter pool according to the evaluation dimension and test scenario type corresponding to each evaluation factor combination to generate multiple agent configuration parameter groups corresponding to each evaluation factor combination includes: Select at least one agent configuration parameter type from the agent configuration parameter pool that matches both the evaluation dimension and the test scenario type corresponding to the target evaluation factor combination; wherein, the target evaluation factor combination is any evaluation factor combination in the evaluation factor combination set; Obtain the parameter value generation rules corresponding to each of the intelligent agent configuration parameter types, and generate multiple parameter values ​​corresponding to each of the intelligent agent configuration parameter types according to the parameter value generation rules; The parameter values ​​are arranged and combined according to the preset permutation and combination rules to generate multiple agent configuration parameter groups corresponding to the target evaluation factor combination.

[0069] This invention, through matching parameter types from a parameter pool, generating parameter values, and arranging and combining them to generate parameter groups, enables a systematic generation process from parameter type to parameter value and then to parameter group, thereby improving the flexibility of parameter configuration and the comprehensiveness of testing.

[0070] Optionally, in this embodiment of the invention, after generating the total evaluation test information based on the set of evaluation factor combinations, the configuration parameter groups of each agent, and the values ​​of each evaluation index, the method further includes: Obtain the first agent configuration parameter group, first evaluation dimension, and first test scenario type corresponding to the verification production command agent; Based on the first intelligent agent configuration parameter group, the first evaluation dimension and the first test scenario type, the first evaluation index value corresponding to the verification production command intelligent agent is matched in the total evaluation test information; The verification production command intelligence agent is evaluated and tested to obtain the second evaluation index value; The first evaluation index value and the second evaluation index value are compared. If the comparison result does not meet the preset conditions, the compensation configuration parameter corresponding to the first evaluation dimension is added to the agent configuration parameter pool.

[0071] This invention, through the introduction of a verification production command agent after generating total evaluation test information to compare actual and predicted values, and updating the parameter pool when conditions are not met, enables the self-verification and continuous optimization of the evaluation system, thereby improving the accuracy and reliability of the total evaluation test information.

[0072] Optionally, in this embodiment of the invention, the step of evaluating and testing the production command agents corresponding to each of the agent configuration parameter groups to obtain the evaluation index values ​​corresponding to each agent configuration parameter group includes: Input multimodal test data matching the target test scenario type into the target production command agent; wherein, the target production command agent is the production command agent corresponding to the second agent configuration parameter group, the second agent configuration parameter group is any agent configuration parameter group in each of the agent configuration parameter groups, and the target test scenario type is any test scenario type in the test scenario type set; Receive the decision response results output by the target production command intelligence agent; The decision response results are quantitatively analyzed according to preset evaluation rules to obtain the evaluation index values ​​corresponding to the second agent configuration parameter group.

[0073] The embodiments of the present invention input multimodal test data into the target intelligent agent and perform quantitative analysis on its decision response results to obtain evaluation index values, thereby enabling objective and accurate quantitative evaluation of the intelligent agent's performance.

[0074] Optionally, in this embodiment of the invention, generating total evaluation test information based on the set of evaluation factor combinations, the configuration parameter groups of each agent, and the values ​​of each evaluation index includes: Based on the configuration parameter groups of each agent corresponding to the target evaluation factor combination, and the evaluation index values ​​corresponding to the configuration parameter groups of each agent corresponding to the target evaluation factor combination, sub-evaluation test information corresponding to the target evaluation factor combination is generated. The sub-evaluation test information corresponding to each of the aforementioned evaluation factors is aggregated to generate total evaluation test information.

[0075] This invention generates sub-evaluation test information by first combining target evaluation factors, and then aggregating all sub-information to generate total evaluation test information. This generates a well-structured total evaluation test information, providing an evaluation benchmark for subsequent agent testing.

[0076] Optionally, in this embodiment of the invention, the step of testing and evaluating the performance of the production command agent to be evaluated based on the total evaluation test information includes: Obtain the configuration parameter group of the third agent, the second evaluation dimension, the second test scenario type, and the maximum evaluation threshold corresponding to the production command agent to be evaluated; Based on the third agent configuration parameter group, the second evaluation dimension, and the second test scenario type, the third evaluation index value corresponding to the production command agent to be evaluated is matched in the total evaluation test information; The third evaluation index value is compared with the maximum evaluation threshold to generate agent evaluation information for the production command agent to be evaluated; wherein, the agent evaluation information is used to indicate whether the configuration parameters of the production command agent to be evaluated should be adjusted.

[0077] This invention, through obtaining the configuration and scenario of the agent to be evaluated, matches index values ​​in the total evaluation test information and compares them with thresholds to generate evaluation information. This enables rapid performance evaluation and qualification judgment of new agents, and provides guidance for parameter adjustment.

[0078] Based on the above method embodiments, a specific embodiment of a production command intelligent agent testing and evaluation method is provided. In this specific embodiment, the system mentioned refers to any system configured with this method.

[0079] In this specific embodiment, the system first performs the operation of obtaining the evaluation dimensions and test scenario types. The system sets the evaluation dimension set. for: ; in, Indicates the first The evaluation dimensions include decision response delay, resource allocation rationality, anomaly recovery capability, multi-objective optimization balance, or knowledge reasoning consistency. The total number of predefined evaluation dimensions for the system. It should be noted that these dimensions are not arbitrarily selected, but rather derived jointly by domain experts and systems engineers based on the core quality attributes of production command operations, ensuring coverage of the key performance requirements that the production command agent must meet in actual operation.

[0080] At the same time, the system sets a set of test scenario types. for: ; in, Indicates the first Various test scenario types, such as sudden equipment failures on production lines, supply chain disruptions, fluctuations in energy supply, influx of high-concurrency orders, or cross-plant collaborative scheduling, etc., are typical or extreme working conditions. The total number of predefined test scenario types for the system. It should be noted that these scenarios are derived from historical production logs, fault case libraries, stress test plans, and future simulations, aiming to cover the full spectrum of operating conditions from steady-state operation to extreme disturbances.

[0081] In this specific embodiment, after obtaining the evaluation dimensions and test scenario types, the system performs a construction operation for the combination of evaluation factors. This specific embodiment uses Cartesian product operations to combine the evaluation dimension set. With test scenario type set By combining each pair of factors, a set of evaluation factor combinations can be generated. : ; Each element This represents an independent combination of evaluation factors, namely, "in the scenario". The next evaluation of the production command intelligence agent in the dimension "Performance on the surface".

[0082] In this specific embodiment, in generating the set of evaluation factor combinations Then, the system performs the operation of generating agent configuration parameter groups. In this specific embodiment, an agent configuration parameter pool is pre-built. : ; Each of them This is a parameter template containing several adjustable configuration items, such as the inference temperature for large models. Knowledge retrieval depth Action space pruning threshold or multi-agent communication frequency And each template Associate an applicable condition This is used to declare which combinations of evaluation factors this template applies to. For any combination of evaluation factors... The system retrieves parameters from the pool. Filter out all that meet the criteria Templates that form subsets : ; This filtering mechanism ensures that only parameter templates semantically matching the current evaluation objective are enabled, preventing invalid or conflicting configurations from interfering with test results. Subsequently, for each The system parses its contents. One adjustable parameter And obtain the parameter value generation rules corresponding to each parameter. These rules can be continuous intervals, discrete sets, or probability distributions. Based on these rules and a preset permutation and combination strategy, the system generates all feasible agent configuration parameter sets for that combination. : ; in, A specific parameter configuration vector represents an instantiable agent variant. The permutation and combination strategy can be full combinatorial, orthogonal experimental design, or Latin hypercube sampling.

[0083] In this specific embodiment, after generating a set of agent configuration parameter groups corresponding to all combinations of evaluation factors, the system performs evaluation tests on the production command agent corresponding to each agent configuration parameter group. This is done for each test scenario type. The system constructs a multimodal test dataset that strictly matches its semantics. : ; in, Indicates the first The multimodal input at each time step includes structured data, unstructured text, images, or time-series signals, and the data content, timing logic, and perturbation strength are all determined according to the scenario. The system synthesizes or samples data according to the defined parameters; wherein, the structured data can be equipment status codes or inventory levels, the unstructured text can be work order descriptions or alarm logs, the image can be a production line monitoring screen, and the timing signal can be a current or temperature curve. Input is configured by the agent's parameter set. Configured production command intelligent agent This production command agent performs reasoning and decision-making based on its internal large model and configuration parameters, and outputs a decision response sequence. : ; in, For the agent in time step The system generates specific decisions, such as executable instructions like "schedule robot R3 to workstation W5" or "pause order O123 and activate backup supplier." Subsequently, the system bases these decisions on the current evaluation dimensions. Call the preset evaluation function For decision sequences Compared with reference standards Perform quantitative comparisons and calculate evaluation index values. : ; in, The reference standard decision sequence can be generated by domain experts' annotation, high-fidelity digital twin simulation, or historical best operation records; As a scalar value, its physical meaning is determined by dimension. Decision, for example when In response to delays, It can be the reciprocal of the average response time; when When it is the resource utilization rate, It can be the ratio of output per unit time to resource consumption.

[0084] In this specific embodiment, after completing the evaluation tests of all agent configuration parameter groups, the system performs an integration operation of the overall evaluation test information. First, for each evaluation factor combination... Construct its corresponding sub-evaluation test information : ; Each element records a specific parameter configuration. and the evaluation index values ​​measured under this combination. Sub-evaluation test information Essentially, it's a local "configuration-performance" mapping table, reflecting the impact of different parameters on performance under fixed dimensions and scenarios. Subsequently, the system merges all sub-information to generate global overall evaluation test information. : ; Overall assessment and testing information It is persistently stored in the form of a database or knowledge graph, supporting multidimensional indexing and efficient querying.

[0085] In this specific embodiment, when a verification production command intelligent agent is introduced, the system obtains its configuration parameter set. Corresponding evaluation dimensions With test scenario type , forming a verification combination The system first in In the query, does it exist? If a matching record exists, extract the predictive indicator value. If not, it is estimated through nearest neighbor combination interpolation. Subsequently, the system performs actual evaluation tests on the verification production command agent to obtain measured index values. The system calculates the deviation between the two and determines whether it exceeds a preset tolerance threshold. That is, to check whether the following conditions are met: ; If the condition is met, it indicates that the current overall evaluation test information contains significant errors or knowledge gaps in this area. In this case, the system determines that the knowledge base needs to be updated: [the data will be updated in conjunction with the dimensions]. Add the corresponding compensation configuration parameters to the agent configuration parameter pool. In the middle, and with The entire process, from parameter generation and testing to information collection, is re-executed around the central element to ultimately generate the updated version. The compensation configuration parameters can be a newly added robustness enhancement module or a new inference strategy.

[0086] In this specific embodiment, the system obtains the configuration parameter set of the production command intelligence agent to be evaluated. Target evaluation dimensions Target test scenario type and the maximum evaluation threshold set by the user The system provides comprehensive evaluation and testing information. Mid-position combination And query or interpolate to obtain the corresponding predictive index value. The system will and The system performs comparisons and generates evaluation conclusions based on the comparison results. Specifically, the system executes the following judgment logic: ; If the conclusion is that optimization is needed, the system can be further analyzed. Extract parameter features from several optimal configurations and generate targeted adjustment suggestions, such as "It is recommended to reduce the inference temperature from 0.8 to 0.6 to improve decision stability" or "It is recommended to enable the cross-plant collaboration module to cope with supply chain disruption scenarios".

[0087] This specific embodiment comprehensively acquires evaluation dimensions and test scenario types, broadly covering actual production situations and avoiding the problem of limited coverage in traditional evaluation scenarios. When generating agent configuration parameter sets, matching parameters are selected from a pre-built parameter pool based on evaluation factor combinations and generated according to rules, improving the flexibility and diversity of parameter generation and enabling dynamic generation of adaptive parameter sets. When acquiring evaluation index values, multimodal test data is input to the production command agent and quantified and analyzed according to preset rules, making the quantification of evaluation indicators more accurate and ensuring the reliability of evaluation results. After generating total evaluation test information, the accuracy of evaluation is improved by verifying the update of the production command agent. For the production command agent to be evaluated, the target evaluation index value is determined based on the total evaluation test information, and the evaluation test result is determined by combining the maximum evaluation threshold, providing accurate guidance for adjusting agent configuration parameters.

[0088] like Figure 2 As shown, based on the above method embodiments, corresponding apparatus embodiments are provided; An embodiment of the present invention provides a production command intelligent agent testing and evaluation device, including: an evaluation factor combination module 201, a configuration parameter group module 202, an evaluation index value module 203, and an evaluation test module 204; The evaluation factor combination module 201 is used to combine any evaluation dimension in the evaluation dimension set with any test scenario type in the test scenario type set to obtain an evaluation factor combination set; wherein, the evaluation factor combination set contains at least one evaluation factor combination. The configuration parameter group module 202 is used to perform parameter matching in a pre-built intelligent agent configuration parameter pool according to the evaluation dimension and test scenario type corresponding to each evaluation factor combination, and generate multiple intelligent agent configuration parameter groups corresponding to each evaluation factor combination. The evaluation index value module 203 is used to evaluate and test the production command intelligent agent corresponding to each intelligent agent configuration parameter group to obtain the evaluation index value corresponding to each intelligent agent configuration parameter group. The evaluation and testing module 204 is used to generate total evaluation and testing information based on the set of evaluation factors, the configuration parameter groups of each agent, and the evaluation index values, and to test and evaluate the production command agent to be evaluated based on the total evaluation and testing information.

[0089] Optionally, in this embodiment of the invention, a condition acquisition submodule is further included before the evaluation factor combination module 201; The condition acquisition submodule is used to acquire an evaluation dimension set and a test scenario type set; wherein, the evaluation dimension set contains at least one evaluation dimension, and the test scenario type set contains at least one test scenario type.

[0090] By obtaining the set of evaluation dimensions and the set of test scenario types, the embodiments of the present invention can clarify the input range and preconditions of the test, and ensure the integrity and executability of subsequent combinations and testing processes.

[0091] Optionally, in this embodiment of the invention, the configuration parameter group module 202 includes: a type matching submodule, a parameter generation submodule, and a parameter combination submodule; The type matching submodule is used to select at least one agent configuration parameter type from the agent configuration parameter pool that matches both the evaluation dimension and the test scenario type corresponding to the target evaluation factor combination; wherein, the target evaluation factor combination is any evaluation factor combination in the evaluation factor combination set; The parameter generation submodule is used to obtain the parameter value generation rules corresponding to each of the intelligent agent configuration parameter types, and generate multiple parameter values ​​corresponding to each of the intelligent agent configuration parameter types according to the parameter value generation rules. The parameter combination submodule is used to arrange and combine the parameter values ​​according to a preset permutation and combination rule to generate multiple agent configuration parameter groups corresponding to the target evaluation factor combination.

[0092] This invention, through matching parameter types from a parameter pool, generating parameter values, and arranging and combining them to generate parameter groups, enables a systematic generation process from parameter type to parameter value and then to parameter group, thereby improving the flexibility of parameter configuration and the comprehensiveness of testing.

[0093] Optionally, in this embodiment of the invention, after the evaluation and testing module 204, the following submodules are further included: a verification parameter submodule, a first indicator matching submodule, an evaluation and testing submodule, and a parameter update submodule; The verification parameter submodule is used to obtain the first intelligent agent configuration parameter group, the first evaluation dimension and the first test scenario type corresponding to the verification production command intelligent agent; The first indicator matching submodule is used to match the first evaluation indicator value corresponding to the verification production command intelligent agent in the total evaluation test information according to the first intelligent agent configuration parameter group, the first evaluation dimension and the first test scenario type; The evaluation and testing submodule is used to evaluate and test the verification production command intelligent agent to obtain the second evaluation index value. The parameter update submodule is used to compare the first evaluation index value and the second evaluation index value. When the comparison result does not meet the preset conditions, the compensation configuration parameter corresponding to the first evaluation dimension is added to the agent configuration parameter pool.

[0094] This invention, through the introduction of a verification production command agent after generating total evaluation test information to compare actual and predicted values, and updating the parameter pool when conditions are not met, enables the self-verification and continuous optimization of the evaluation system, thereby improving the accuracy and reliability of the total evaluation test information.

[0095] Optionally, in this embodiment of the invention, the evaluation index value module 203 includes: a data input submodule, a decision response submodule, and a quantitative analysis submodule; The data input submodule is used to input multimodal test data that matches the target test scenario type to the target production command agent; wherein, the target production command agent is the production command agent corresponding to the second agent configuration parameter group, the second agent configuration parameter group is any one of the agent configuration parameter groups, and the target test scenario type is any one of the test scenario types in the test scenario type set; The decision response submodule is used to receive the decision response results output by the target production command intelligence agent; The quantitative analysis submodule is used to perform quantitative analysis on the decision response results according to preset evaluation rules to obtain the evaluation index values ​​corresponding to the second agent configuration parameter group.

[0096] The embodiments of the present invention input multimodal test data into the target intelligent agent and perform quantitative analysis on its decision response results to obtain evaluation index values, thereby enabling objective and accurate quantitative evaluation of the intelligent agent's performance.

[0097] Optionally, in this embodiment of the invention, the evaluation test module 204 includes: a sub-evaluation test information sub-module and a total evaluation test information sub-module; The sub-evaluation test information submodule is used to generate sub-evaluation test information corresponding to the target evaluation factor combination based on the configuration parameter groups of each agent corresponding to the target evaluation factor combination and the evaluation index values ​​corresponding to the configuration parameter groups of each agent corresponding to the target evaluation factor combination. The overall evaluation test information submodule is used to aggregate the sub-evaluation test information corresponding to each of the evaluation factors to generate overall evaluation test information.

[0098] This invention generates sub-evaluation test information by first combining target evaluation factors, and then aggregating all sub-information to generate total evaluation test information. This generates a well-structured total evaluation test information, providing an evaluation benchmark for subsequent agent testing.

[0099] Optionally, in this embodiment of the invention, the evaluation and testing module 204 further includes: an evaluation parameter submodule, a second index matching submodule, and an evaluation information submodule; The evaluation parameter submodule is used to obtain the third agent configuration parameter group, second evaluation dimension, second test scenario type and maximum evaluation threshold corresponding to the production command agent to be evaluated; The second indicator matching submodule is used to match the third evaluation indicator value corresponding to the production command intelligent agent to be evaluated in the total evaluation test information according to the third intelligent agent configuration parameter group, the second evaluation dimension and the second test scenario type. The evaluation information submodule is used to compare the third evaluation index value with the maximum evaluation threshold to generate agent evaluation information for the production command agent to be evaluated; wherein, the agent evaluation information is used to indicate whether the configuration parameters of the production command agent to be evaluated should be adjusted.

[0100] This invention, through obtaining the configuration and scenario of the agent to be evaluated, matches index values ​​in the total evaluation test information and compares them with thresholds to generate evaluation information. This enables rapid performance evaluation and qualification judgment of new agents, and provides guidance for parameter adjustment.

[0101] It is understood that the above-described device embodiments correspond to the method embodiments of the present invention, and can implement the production command intelligent agent testing and evaluation method provided by any of the above-described method embodiments of the present invention.

[0102] In this embodiment of the invention, the evaluation factor combination module 201 combines any evaluation dimension from the evaluation dimension set with any test scenario type from the test scenario type set, thereby constructing a comprehensive set of evaluation factor combinations covering different evaluation perspectives and test environments, laying the foundation for subsequent systematic testing. The configuration parameter group module 202 matches parameters in a pre-built agent configuration parameter pool based on the evaluation dimension and test scenario type corresponding to each evaluation factor combination, dynamically generating diverse agent configurations for different evaluation factor combinations, improving the relevance and coverage of the tests. The evaluation index value module 203 evaluates and tests the production command agents corresponding to each agent configuration parameter group, obtaining specific performance data of agents under different configurations in various scenarios, achieving refined performance measurement. The evaluation test module 204 generates total evaluation test information based on the evaluation factor combination set, each agent configuration parameter group, and each evaluation index value, and tests and evaluates the production command agent to be evaluated based on this total evaluation test information, integrating all test data to form a comprehensive evaluation benchmark, and using this benchmark to evaluate the performance of the agent to be evaluated.

[0103] It should be noted that the device embodiments described above are merely illustrative, and some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can specifically be implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0104] Based on the above-described embodiment of a production command intelligent agent testing and evaluation method, another embodiment of the present invention provides a terminal device, which includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements a production command intelligent agent testing and evaluation method according to any embodiment of the present invention.

[0105] For example, in this embodiment, the computer program can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the terminal device.

[0106] The terminal device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal device may include, but is not limited to, a processor and memory.

[0107] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the terminal device, connecting all parts of the terminal device via various interfaces and lines.

[0108] Based on the above-described method embodiments, another embodiment of the present invention provides a computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to execute the production command intelligent agent testing and evaluation method described in any of the above-described method embodiments of the present invention.

[0109] The modules / units integrated in the device / terminal equipment, if implemented as software functional units and sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0110] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for testing and evaluating a production command intelligent agent, characterized in that, include: Each evaluation dimension in the evaluation dimension set is combined with any test scenario type in the test scenario type set to obtain an evaluation factor combination set; wherein, the evaluation factor combination set contains at least one evaluation factor combination. Based on the evaluation dimension and test scenario type corresponding to each combination of evaluation factors, parameter matching is performed in the pre-built agent configuration parameter pool to generate multiple agent configuration parameter groups corresponding to each combination of evaluation factors. The production command intelligent agents corresponding to each of the intelligent agent configuration parameter groups are evaluated and tested to obtain the evaluation index values ​​corresponding to each of the intelligent agent configuration parameter groups. Based on the set of evaluation factors, the configuration parameter groups of each agent, and the values ​​of each evaluation index, a total evaluation test information is generated, and the production command agent to be evaluated is tested and evaluated based on the total evaluation test information.

2. The method for testing and evaluating a production command intelligent agent as described in claim 1, characterized in that, Before combining any evaluation dimension from the evaluation dimension set with any test scenario type from the test scenario type set to obtain the evaluation factor combination set, the method further includes: Obtain an evaluation dimension set and a test scenario type set; wherein the evaluation dimension set contains at least one evaluation dimension, and the test scenario type set contains at least one test scenario type.

3. The method for testing and evaluating a production command intelligent agent as described in claim 1, characterized in that, The step involves matching parameters in a pre-built agent configuration parameter pool based on the evaluation dimension and test scenario type corresponding to each evaluation factor combination, generating multiple agent configuration parameter groups corresponding to each evaluation factor combination, including: Select at least one agent configuration parameter type from the agent configuration parameter pool that matches both the evaluation dimension and the test scenario type corresponding to the target evaluation factor combination; wherein, the target evaluation factor combination is any evaluation factor combination in the evaluation factor combination set; Obtain the parameter value generation rules corresponding to each of the intelligent agent configuration parameter types, and generate multiple parameter values ​​corresponding to each of the intelligent agent configuration parameter types according to the parameter value generation rules; The parameter values ​​are arranged and combined according to the preset permutation and combination rules to generate multiple agent configuration parameter groups corresponding to the target evaluation factor combination.

4. The method for testing and evaluating a production command intelligent agent as described in claim 1, characterized in that, After generating the total evaluation test information based on the set of evaluation factor combinations, the configuration parameter groups of each agent, and the values ​​of each evaluation index, the method further includes: Obtain the first agent configuration parameter group, first evaluation dimension, and first test scenario type corresponding to the verification production command agent; Based on the first intelligent agent configuration parameter group, the first evaluation dimension and the first test scenario type, the first evaluation index value corresponding to the verification production command intelligent agent is matched in the total evaluation test information; The verification production command intelligence agent is evaluated and tested to obtain the second evaluation index value; The first evaluation index value and the second evaluation index value are compared. If the comparison result does not meet the preset conditions, the compensation configuration parameter corresponding to the first evaluation dimension is added to the agent configuration parameter pool.

5. The method for testing and evaluating a production command intelligent agent as described in claim 1, characterized in that, The evaluation and testing of the production command intelligent agents corresponding to each of the intelligent agent configuration parameter groups to obtain the evaluation index values ​​corresponding to each of the intelligent agent configuration parameter groups includes: Input multimodal test data matching the target test scenario type into the target production command agent; wherein, the target production command agent is the production command agent corresponding to the second agent configuration parameter group, the second agent configuration parameter group is any agent configuration parameter group in each of the agent configuration parameter groups, and the target test scenario type is any test scenario type in the test scenario type set; Receive the decision response results output by the target production command intelligence agent; The decision response results are quantitatively analyzed according to preset evaluation rules to obtain the evaluation index values ​​corresponding to the second agent configuration parameter group.

6. The method for testing and evaluating a production command intelligent agent as described in claim 3, characterized in that, The step of generating total evaluation test information based on the set of evaluation factor combinations, the configuration parameter groups of each agent, and the values ​​of each evaluation index includes: Based on the configuration parameter groups of each agent corresponding to the target evaluation factor combination, and the evaluation index values ​​corresponding to the configuration parameter groups of each agent corresponding to the target evaluation factor combination, sub-evaluation test information corresponding to the target evaluation factor combination is generated. The sub-evaluation test information corresponding to each of the aforementioned evaluation factors is aggregated to generate total evaluation test information.

7. The method for testing and evaluating a production command intelligent agent as described in claim 1, characterized in that, The process of testing and evaluating the performance of the production command agent to be evaluated based on the overall evaluation test information includes: Obtain the configuration parameter group of the third agent, the second evaluation dimension, the second test scenario type, and the maximum evaluation threshold corresponding to the production command agent to be evaluated; Based on the third agent configuration parameter group, the second evaluation dimension, and the second test scenario type, the third evaluation index value corresponding to the production command agent to be evaluated is matched in the total evaluation test information; The third evaluation index value is compared with the maximum evaluation threshold to generate agent evaluation information for the production command agent to be evaluated; wherein, the agent evaluation information is used to indicate whether the configuration parameters of the production command agent to be evaluated should be adjusted.

8. A testing and evaluation device for a production command intelligent agent, characterized in that, include: The evaluation factor combination module, the configuration parameter group module, the evaluation index value module, and the evaluation test module are included. The evaluation factor combination module is used to combine any evaluation dimension in the evaluation dimension set with any test scenario type in the test scenario type set to obtain an evaluation factor combination set; wherein, the evaluation factor combination set contains at least one evaluation factor combination. The configuration parameter group module is used to perform parameter matching in a pre-built agent configuration parameter pool according to the evaluation dimension and test scenario type corresponding to each evaluation factor combination, and generate multiple agent configuration parameter groups corresponding to each evaluation factor combination. The evaluation index value module is used to evaluate and test the production command intelligent agent corresponding to each intelligent agent configuration parameter group to obtain the evaluation index value corresponding to each intelligent agent configuration parameter group. The evaluation and testing module is used to generate total evaluation and testing information based on the set of evaluation factors, the configuration parameter groups of each agent, and the evaluation index values, and to test and evaluate the production command agent to be evaluated based on the total evaluation and testing information.

9. A terminal device, characterized in that, The system includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, it implements a production command agent testing and evaluation method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, include: A stored computer program, wherein, when the computer program is executed, it controls the device containing the computer-readable storage medium to perform a production command agent testing and evaluation method as described in any one of claims 1-7.

Images