A method and device for evaluating the capabilities of a large model-based simulated agent

Abstract

The application discloses a capability evaluation method and device for a simulation agent based on a large model, the method comprising: obtaining simulation agent configuration data information and evaluation scheme data information; performing comprehensive evaluation processing on the simulation agent configuration data information and the evaluation scheme data information in three dimensions of basic capability, application capability and emergent capability; obtaining capability evaluation data information; and performing comprehensive calculation processing on the capability evaluation data information to obtain target capability grade classification result information.

Description

Technical Field

[0001] This invention relates to the field of simulation technology, and in particular to a method and apparatus for evaluating the capabilities of simulated intelligent agents based on large models. Background Technology

[0002] With the rapid development of artificial intelligence technology, Large Language Model (LLM) technology has become increasingly mature. In the field of simulation, by integrating LLM technology with simulation model building technology, it is possible to construct simulation agents based on large models. Simulation agents based on large models have broad application prospects in adversarial simulation, situational analysis, and assisted computation, and their performance directly affects the overall performance of the simulation system and the reliability of the simulation results. However, how to comprehensively and accurately evaluate the operational capabilities of simulation agents based on large models has become an urgent technical problem to be solved.

[0003] Existing technologies for detecting the capabilities of simulated intelligent agents mainly include detection methods based on attribute data, detection methods based on runtime data, detection methods based on task completion, and detection methods based on a single dimension. However, these existing technologies have the following technical limitations: Attribute data-based detection methods can only detect static configuration parameters and cannot reflect the actual response performance of the agent software module in a dynamic operating environment; runtime data-based detection methods rely on pre-defined formatted data interfaces, making it difficult to adapt to the natural language processing interface characteristics of large-model simulated intelligent agents, thus preventing the collection of agent response data through natural language question-and-answer methods; task completion-based detection methods struggle to detect the communication efficiency and resource scheduling performance of the agent in multi-agent communication scenarios; and single-dimensional detection methods cannot comprehensively detect the operating status of multiple software modules, such as the stability of the memory module, the response speed of the inference module, and the transmission efficiency of the communication module. Furthermore, none of these methods can effectively monitor performance degradation and state drift issues in the agent software system during long-term operation. Therefore, existing technologies cannot comprehensively detect the operating capabilities of large-model simulated intelligent agent software systems and cannot provide effective technical basis for the optimization and improvement of the software system. Summary of the Invention

[0004] The technical problem to be solved by this invention is to provide a method and apparatus for evaluating the capabilities of simulated intelligent agents based on large models. This solves the technical problem that existing detection technologies are unable to comprehensively detect the operational capabilities of simulated intelligent agent software systems based on large models. As a result, it is beneficial to comprehensively and accurately evaluate the operational capabilities of simulated intelligent agent software systems, discover the performance bottlenecks and operational defects of the software systems, provide quantitative basis for the optimization and improvement of the software systems, and improve the reliability of the simulation systems and the accuracy of the simulation results.

[0005] To address the aforementioned technical problems, a first aspect of this invention discloses a method for evaluating the capabilities of a simulated intelligent agent based on a large model, the method comprising: S1, acquire configuration data information and evaluation scheme data information of the simulated intelligent agent; S2, process the configuration data information of the simulated intelligent agent and the evaluation scheme data information to obtain capability assessment data information; the capability assessment data information includes basic capability assessment data information, application capability assessment data information and emergent capability assessment data information; S3, perform comprehensive calculation and processing on the capability assessment data to obtain the target capability level classification result information.

[0006] As an optional implementation, in the first aspect of the present invention, the processing of the configuration data information of the simulated intelligent agent and the evaluation scheme data information to obtain capability assessment data information includes: S21, perform basic capability evaluation processing on the configuration data information of the simulated intelligent agent and the evaluation scheme data information to obtain basic capability evaluation data information; S22, The application capability evaluation data information of the simulated intelligent agent configuration data information and the evaluation scheme data information are processed to obtain application capability evaluation data information; S23, perform emergent capability evaluation processing on the configuration data information of the simulated intelligent agent and the evaluation scheme data information to obtain emergent capability evaluation data information.

[0007] As an optional implementation, in the first aspect of the present invention, the basic capability assessment processing of the configuration data information of the simulated intelligent agent and the evaluation scheme data information to obtain basic capability evaluation data information includes: S211, perform initial question-and-answer evaluation processing on the configuration data information of the simulated intelligent agent and the evaluation scheme data information to obtain initial evaluation result data information; S212, Process the configuration data information of the simulated intelligent agent to obtain simulation operation record data information; S213, perform a second evaluation process on the configuration data information of the simulated intelligent agent and the evaluation scheme data information to obtain the second evaluation result data information; S214, perform matching degree calculation on the initial evaluation result data information, the second evaluation result data information, and the simulation intelligent agent configuration data information to obtain basic capability evaluation data information.

[0008] A second aspect of this invention discloses a capability evaluation device for a simulated intelligent agent based on a large model, the device comprising: The acquisition module is used to acquire configuration data information and evaluation scheme data information of the simulated intelligent agent; The capability assessment module is used to process the configuration data information of the simulated intelligent agent and the assessment scheme data information to obtain capability assessment data information; the capability assessment data information includes basic capability assessment data information, application capability assessment data information and emergent capability assessment data information. The comprehensive calculation module is used to perform comprehensive calculation and processing on the capability assessment data to obtain the target capability level classification result information.

[0009] A third aspect of this invention discloses another capability evaluation device for simulated intelligent agents based on large models, the device comprising: processor; A memory coupled to the processor stores executable program code; The processor calls the executable program code stored in the memory to execute some or all of the steps of the capability evaluation method for a large-model-based simulated intelligent agent disclosed in the first aspect of the present invention.

[0010] The fourth aspect of this invention discloses a computer-readable storage medium storing computer instructions, which, when invoked, are used to execute some or all of the steps of the capability assessment method for a large-model-based simulated intelligent agent disclosed in the first aspect of this invention.

[0011] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: In this embodiment of the invention, by acquiring the configuration data information and evaluation scheme data information of the simulated intelligent agent, the configuration data information and evaluation scheme data information of the simulated intelligent agent are comprehensively evaluated and processed in three dimensions: basic capability, application capability and emergent capability, to obtain capability assessment data information. The capability assessment data information is then comprehensively calculated and processed to obtain the target capability level classification result information. It can be seen that this application has the following technical effects: (1) By calling the API interface of the large language model to conduct natural language question answering evaluation, it is possible to collect natural language response data of the intelligent agent software system, which solves the technical limitation of traditional detection methods that can only collect formatted data, and can detect the data drift and state decay problem of the intelligent agent memory module during long-term operation; (2) By conducting parallel simulation of multiple intelligent agents through the system simulation system, it is possible to collect the operation data of the intelligent agents in a controlled virtual environment, avoiding the risks brought about by testing in the actual system; (3) The method of intelligence testing is used to judge, compare and evaluate the changes in the ability of the simulated intelligent agent before and after "practice", and the emergence ability of the simulated intelligent agent is directly quantitatively evaluated by the data of the simulated intelligent agent's autonomous system construction and system information transmission. The basic ability, application ability and emergence ability of the simulated intelligent agent are evaluated and calculated through numerical calculation, providing a basis for the improvement and evolution of the large model simulated intelligent agent. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart illustrating a capability assessment method for a large-model-based simulated intelligent agent disclosed in an embodiment of the present invention. Figure 2 This is a schematic diagram of the structure of a capability evaluation device for a large-model-based simulated intelligent agent disclosed in an embodiment of the present invention; Figure 3 This is a schematic diagram of another capability evaluation device for a large-model-based simulated intelligent agent disclosed in an embodiment of the present invention. Detailed Implementation

[0014] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0015] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0016] Specific features, structures, or characteristics described in connection with the embodiments may be included in at least one embodiment of the invention. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments.

[0017] Example 1 Please see Figure 1 , Figure 1 This is a flowchart illustrating a capability evaluation method for a large-model-based simulated intelligent agent, as disclosed in an embodiment of the present invention. Wherein, Figure 1 The described method for evaluating the capabilities of simulated intelligent agents is applied to simulation evaluation systems, such as local servers or cloud servers used for game deduction and simulation evaluation; however, this embodiment of the invention is not limited to these applications. Figure 1 As shown, the method may include the following operations: S1, acquire configuration data information and evaluation scheme data information of the simulated intelligent agent.

[0018] It should be noted that the simulated agent configuration data information refers to the initialization configuration data of the simulated agent software system built based on a large model. This data is stored in a structured data format in the configuration database and includes agent instance type information, agent function configuration information, agent operating parameter information, and agent logic mapping information. Specifically, the agent instance type information includes the instance name string, instance type code, and instance classification description text; the agent function configuration information includes function description text, task scope parameters, and interface boundary configurations; the agent operating parameter information includes numerical configuration parameters such as movement speed, communication bandwidth, output frequency, response latency, and acquisition accuracy; and the agent logic mapping information includes input / output mapping configuration, communication interface configuration, and execution sequence configuration. The evaluation scheme data information refers to the capability evaluation configuration parameter data, stored in a structured data format in the evaluation database, including the evaluation information set, preset scenario configuration information, and scoring threshold parameter information. The assessment scheme data is stored and managed through a database system. The assessment information set includes the text content and format configuration of the assessment questions; the preset scenario configuration information includes the parameter configuration and handling rules configuration of the simulation scenario; and the scoring threshold parameter information includes numerical threshold parameters used for capability detection. The above configuration parameter data can be set and updated through configuration files, database tables, or API interfaces, etc. Specifically, this embodiment of the invention does not limit the specific methods used.

[0019] It should be noted that the simulation intelligent agent based on the large model refers to a software system that integrates large language model technology with simulation model construction technology. This software system includes the following functional modules: data acquisition module (for collecting input data from the external environment), data storage module (for writing data into memory or database), data retrieval module (for querying historical data from storage), model inference module (for calling the large language model to perform inference calculations), sequence generation module (for generating execution sequences), and instruction output module (for outputting execution instructions).

[0020] It should be noted that the method described in this invention is executed on a computer system, and the processor calls program code in memory to detect the operational capability of the simulated intelligent agent software system. The simulated intelligent agent runs in a system simulation system, which is a computer-based distributed simulation software platform that uses a discrete event simulation engine to drive the parallel operation and network communication of multiple intelligent agent software instances in a virtual environment. The technical architecture of the system simulation system includes: a simulation engine layer (responsible for time stepping and event scheduling), an intelligent agent runtime layer (responsible for the lifecycle management of intelligent agent software instances), a communication middleware layer (responsible for message passing between intelligent agents), and a data storage layer (responsible for persistent storage of simulation records).

[0021] S2, the configuration data information of the simulated intelligent agent and the evaluation scheme data information are processed to obtain capability assessment data information; the capability assessment data information includes basic capability assessment data information, application capability assessment data information and emergent capability assessment data information.

[0022] It should be noted that the above processing refers to a comprehensive evaluation of the operational capabilities of the simulated intelligent agent software system using a multi-dimensional assessment method, including three dimensions: basic capability evaluation, application capability evaluation, and emergent capability evaluation. The basic capability information refers to the simulated intelligent agent's ability to acquire its own instance type and functional configuration information, as well as its ability to maintain consistent and coherent memory information that aligns with its instance type and functional configuration. The application capability information refers to the simulated intelligent agent's ability to produce inference outputs and execution behaviors appropriate to its own functional configuration in typical scenarios. The emergent capability information refers to the simulated intelligent agent's ability to produce inference outputs and execution behaviors in response to new situations across different scenarios.

[0023] S3, perform comprehensive calculation and processing on the capability assessment data to obtain the target capability level classification result information.

[0024] It should be noted that the target capability level classification result information includes comprehensive capability detection value and capability level information.

[0025] As can be seen, the simulation agent capability evaluation method described in the embodiments of the present invention can comprehensively test the operating performance of the memory module, inference module, execution module and communication module of the simulation agent software system, and realize comprehensive monitoring and capability testing of the operating status of the simulation agent software system.

[0026] In an optional embodiment, the processing of the simulated intelligent agent configuration data information and the evaluation scheme data information to obtain capability assessment data information includes: S21, perform basic capability assessment processing on the configuration data information of the simulated intelligent agent and the data information of the evaluation scheme to obtain basic capability assessment data information.

[0027] It should be noted that the basic capability assessment data includes type configuration matching degree value, parameter configuration matching degree value, logical configuration matching degree value, data consistency detection value, and basic capability comprehensive detection value, all of which are floating-point numerical data.

[0028] S22, the application capability evaluation data information of the simulated intelligent agent configuration data information and the evaluation scheme data information are processed to obtain application capability evaluation data information.

[0029] It should be noted that the application capability evaluation process refers to loading a simulated intelligent agent software instance into a simulation scenario with preset response rules through a system simulation system for parallel operation. This involves collecting the agent's inference output record data and execution record data, and using a pattern matching algorithm to perform conformity checks on the inference output records and an instruction comparison algorithm to perform consistency checks on the execution records. The application capability evaluation data includes output conformity detection values, execution consistency detection values, and comprehensive application capability detection values, all of which are floating-point numerical data. This application capability evaluation process is used to detect the response speed and output quality of the agent's inference module, and the instruction parsing accuracy and execution reliability of the execution module.

[0030] S23, perform emergent capability evaluation processing on the configuration data information of the simulated intelligent agent and the evaluation scheme data information to obtain emergent capability evaluation data information.

[0031] It should be noted that the emergence capability assessment data includes open scenario output compliance value, connection construction detection value, information diffusion detection value, and comprehensive emergence capability detection value, all of which are floating-point numerical data. The emergence capability evaluation process is used to detect the reasoning output and execution behavior capabilities of the simulated agent in response to new situations across scenarios, including the effectiveness of the simulated agent's information acquisition, connection construction, and reasoning output in scenarios without explicit response rules.

[0032] It should be noted that the open scenario output compliance value refers to the compliance of the simulated agent's decisions and behaviors in an open scenario (a simulation scenario where there are no explicit response rules given to the simulated agent). Its value is the ratio of the number of outputs conforming to the basic mapping logic to the total number of outputs, used to quantify the compliance of the inference module's outputs in a dynamic environment. The connection construction detection value refers to the change in network connection density between agents before and after simulation. Its value is the ratio of the connection density after simulation to the connection density before simulation, used to quantify the self-organizing network performance and network topology evolution of the communication module. The information diffusion detection value refers to the propagation coverage of important event messages in the agent network. Its value is the average proportion of simulated agents reached by each important event message, used to quantify the message routing efficiency and network transmission performance of the communication module. The emergent capability comprehensive detection value is used to quantify the overall operational performance and inter-module interface coordination of the simulated agent's communication module and data exchange module under open scenario conditions.

[0033] It is evident that the simulated intelligent agent capability evaluation method described in the embodiments of the present invention can comprehensively test the operational performance of the simulated intelligent agent software system through a comprehensive test of three dimensions: basic capability evaluation, application capability evaluation, and emergent capability evaluation.

[0034] In another optional embodiment, the basic capability assessment processing of the simulated intelligent agent configuration data information and the evaluation scheme data information to obtain basic capability evaluation data information includes: S211, perform initial question-and-answer evaluation processing on the configuration data information of the simulated intelligent agent and the evaluation scheme data information to obtain initial evaluation result data information.

[0035] It should be noted that the initial question-and-answer evaluation process refers to sending natural language question-and-answer requests to the agent by calling the large language model API interface after the simulation agent software system is started and before it runs in any simulation scenario, and collecting the agent's response text data. The initial evaluation result data information includes initial type configuration response information, initial parameter configuration response information, and initial logic configuration response information, all of which are stored in the evaluation result database in text string format.

[0036] It should be noted that the question-and-answer assessment is implemented by calling the large language model API interface via HTTP / HTTPS protocol. The assessment request is encapsulated in JSON format, containing assessment prompts and question-and-answer questions. The above question-and-answer assessment processing is implemented by calling the large language model API interface. Specifically, API services of large language models such as GPT, DeepSeek, and Kimi K2 can be used, but this embodiment of the invention is not limited to any particular type.

[0037] S212, The configuration data information of the simulated intelligent agent is processed to obtain the simulation operation record data information.

[0038] It should be noted that the above processing refers to the preset scenario simulation operation processing. This involves loading a simulation agent software instance into a simulation scenario with preset handling rules through a simulation system. The simulation engine advances the simulation time in a discrete event-driven manner. At each simulation time step, the agent software instance receives environmental state data, calls the inference module within the simulation agent to generate inference output records, and calls the execution module within the simulation agent to output execution records. The simulation operation record data includes the inference output records, execution records, and interaction records of the simulation agent during the simulation process, stored in a structured data format in the simulation record database. Each record contains fields such as timestamp, agent identifier, record type, and record content.

[0039] S213, perform a second evaluation process on the configuration data information of the simulated intelligent agent and the evaluation scheme data information to obtain the second evaluation result data information.

[0040] It should be noted that the second evaluation process refers to evaluating the simulated agent again using the same question-and-answer questions as the initial evaluation after the simulated agent has participated in the simulation of a preset scenario, with the same processing algorithm. The second evaluation result data includes confirmation type configuration response information, confirmation parameter configuration response information, and confirmation logic configuration response information.

[0041] S214, perform matching degree calculation on the initial evaluation result data information, the second evaluation result data information, and the simulation intelligent agent configuration data information to obtain basic capability evaluation data information.

[0042] As can be seen, by implementing the simulation agent capability evaluation method described in the embodiments of the present invention, the basic capabilities and data consistency of the simulation agent can be comprehensively detected through comparative evaluation before and after simulation operation.

[0043] In another optional embodiment, the initial question-and-answer evaluation processing of the simulated intelligent agent configuration data information and the evaluation scheme data information to obtain initial evaluation result data information includes: S2111, preprocess the evaluation information set in the evaluation scheme data information to obtain a preprocessed evaluation information set.

[0044] It should be noted that the preprocessing refers to formatting the question-and-answer assessment questions through a text processing program, including: (1) calling a word segmentation program to segment the assessment question text to obtain word sequence data; (2) calling a stop word filtering program to remove meaningless stop words; (3) calling a template filling program to fill the agent configuration parameters into the placeholder positions in the assessment question template; and (4) calling a format conversion program to convert the assessment questions into the JSON format required by the large language model API interface. The preprocessed assessment information set is stored in a structured data format, including fields such as assessment number information, assessment type information, assessment information, and expected answer format information.

[0045] S2112, the preprocessed evaluation information set and the simulated intelligent agent configuration data information are processed to obtain evaluation prompt word data information.

[0046] It should be noted that the above-mentioned construction process refers to assembling the preprocessed evaluation information set and the simulated agent configuration data into input prompts for the large language model API interface through prompt word engineering technology. The preprocessed evaluation information set includes system prompts (defining the evaluation task background and answer format requirements), context prompts (providing agent configuration information as a reference), and evaluation prompts (specific evaluation questions). The prompt word construction process is implemented through a string concatenation program and a template rendering program; specifically, this embodiment of the invention does not limit the scope of the application.

[0047] S2113, using the evaluation prompt word data information and the large language model, the simulation agent corresponding to the simulation agent configuration data information is processed to obtain the original response data information.

[0048] It should be noted that the above processing refers to calling the large language model API interface via HTTP / HTTPS protocol to send the assessment prompt word data information to the large language model server, and receiving and parsing the response message returned by the server. The original response data information includes fields such as response text content, response timestamp, and response status code. The above processing can use API services of large language models such as GPT, DeepSeek, and Kimi K2, and this embodiment of the invention is not limited thereto.

[0049] S2114, The original response data information is processed by response parsing to obtain the initial evaluation result data information.

[0050] It should be noted that the response parsing process refers to performing structured parsing of the raw response text returned by the large language model using a text parsing program to extract the answer content corresponding to each assessment question. The initial assessment result data information includes initial type configuration response information, initial parameter configuration response information, and initial logic configuration response information.

[0051] It should be noted that the above response parsing and processing can be performed using JSON parsing programs, regular expression matching programs, or data cleaning programs, etc. The specific implementation of this invention is not limited.

[0052] As can be seen, by implementing the simulation agent capability assessment method described in the embodiments of the present invention, through the complete process of test question preprocessing, prompt word construction, API call and response parsing, the question-and-answer response data of the simulation agent can be collected in a standardized manner, thereby improving the quality and consistency of the test data.

[0053] In an optional embodiment, the matching degree calculation processing of the initial evaluation result data information, the second evaluation result data information, and the simulated intelligent agent configuration data information to obtain basic capability assessment data information includes: S2141, perform type configuration matching degree calculation processing on the initial type configuration response information in the initial evaluation result data information, the confirmation type configuration response information in the second evaluation result data information, and the simulation intelligent agent configuration data information to obtain the type configuration matching degree value.

[0054] S2142, perform parameter configuration matching degree calculation on the initial parameter configuration response information in the initial evaluation result data information, the confirmation parameter configuration response information in the second evaluation result data information, and the simulation intelligent agent configuration data information to obtain the parameter configuration matching degree value.

[0055] S2143, perform logic configuration matching degree calculation processing on the initial logic configuration response information in the initial evaluation result data information and the confirmation logic configuration response information in the second evaluation result data information to obtain the logic configuration matching degree value.

[0056] S2144, Perform a consistency comparison process on the initial evaluation result data information and the second evaluation result data information to obtain a data consistency detection value.

[0057] It should be noted that the consistency comparison processing refers to comparing the differences between the two response texts of the simulated agent and calculating the coverage of the post-simulation response text to the pre-simulation response text. Higher coverage indicates better data storage consistency and less data drift in the simulated agent. The above consistency comparison processing is implemented through a computer program, and can specifically use text difference calculation algorithms, coverage calculation algorithms, or text similarity algorithms, etc., without limitation in this embodiment of the invention.

[0058] S2145, calculate and process the type configuration matching degree value, the parameter configuration matching degree value, the logical configuration matching degree value and the data consistency detection value to obtain the basic capability comprehensive detection value.

[0059] It should be noted that the above calculations can be performed using a basic capability comprehensive evaluation calculation model, or using weighted summation algorithms, multi-factor fusion algorithms, comprehensive scoring algorithms, or statistical analysis algorithms. Specifically, the embodiments of the present invention do not limit the specific methods used.

[0060] The comprehensive assessment calculation model for basic capabilities is as follows: ; ; ; In the formula, This is the comprehensive test value of the basic capabilities. Configure a matching degree value for the type. Configure a matching degree value for the parameter. Configure a matching degree value for the logic. The data consistency detection value is... , and These are the first weight parameter, the second weight parameter, and the third weight parameter, respectively.

[0061] It should be noted that the basic capability comprehensive evaluation calculation model is used to comprehensively quantify the operational performance of the simulated intelligent agent's memory module. The data consistency detection value... As a multiplication factor, it reflects the modulating effect of memory module stability on overall performance: when data consistency is high, A value close to 1 indicates that the memory module stores data stably during simulation, with no data loss or tampering; when data consistency is low, A decrease in the value indicates a software fault in the memory module, such as data drift, memory leak, or data overwriting, requiring technical investigation and repair.

[0062] It should be noted that the aforementioned weight parameters are stored in the configuration database and can be set by the user through the configuration interface, or automatically optimized by the system based on detection scene characteristics or historical detection data using machine learning algorithms. Specifically, this embodiment of the invention does not impose limitations. For example, , , .

[0063] S2146, add the type configuration matching degree value, the parameter configuration matching degree value, the logical configuration matching degree value, the data consistency detection value, and the basic capability comprehensive detection value to the basic capability assessment data information to obtain the basic capability assessment data information.

[0064] As can be seen, by implementing the simulation agent capability evaluation method described in the embodiments of the present invention, the data storage stability and reading accuracy of the simulation agent memory module can be effectively detected through semantic similarity calculation and data consistency detection.

[0065] In another optional embodiment, the step of calculating the type configuration matching degree of the initial type configuration response information in the initial evaluation result data information, the confirmation type configuration response information in the second evaluation result data information, and the simulation agent configuration data information to obtain a type configuration matching degree value includes: S21411, Perform a first comparison process on the initial type configuration response information and the simulation agent configuration data information in the initial evaluation result data information to obtain a first initial matching degree value.

[0066] It should be noted that the first comparison process refers to comparing the response text of the simulated intelligent agent regarding instance type, functional configuration, and logical mapping with the initialization configuration text using a semantic similarity calculation algorithm, and calculating the semantic similarity score between the texts. The above comparison process can be performed using methods such as cosine similarity algorithm, BERT semantic similarity algorithm, or large language model API detection, and this embodiment of the invention does not impose any limitations.

[0067] S21412, perform a first comparison process on the confirmation type configuration response information in the second evaluation result data information and the simulation intelligent agent configuration data information to obtain a first confirmation matching degree value.

[0068] It should be noted that the calculation method of the first confirmation matching degree value is the same as that of the first initial matching degree value. The confirmation type configuration response information and the initialization configuration text are compared and processed by a semantic similarity calculation algorithm.

[0069] S21413, the first initial matching degree value and the first confirmed matching degree value are averaged to obtain the type configuration matching degree value.

[0070] It should be noted that the average calculation process refers to adding the first initial matching degree value and the first confirmed matching degree value and then dividing by 2 to obtain the type configuration matching degree value.

[0071] It is evident that implementing the simulation agent capability evaluation method described in the embodiments of the present invention can quantitatively detect the accuracy and consistency of the simulation agent's memory of type configuration information.

[0072] In another optional embodiment, the step of calculating the parameter configuration matching degree of the initial parameter configuration response information in the initial evaluation result data information, the confirmation parameter configuration response information in the second evaluation result data information, and the simulation agent configuration data information to obtain a parameter configuration matching degree value includes: S21421, Perform a second comparison process on the initial parameter configuration response information and the simulation agent configuration data information in the initial evaluation result data information to obtain a second initial matching degree value.

[0073] It should be noted that the second comparison process refers to comparing the response text of the simulated intelligent agent regarding movement speed parameters, communication bandwidth parameters, output frequency parameters, response delay parameters, and acquisition accuracy parameters with the initial configuration content, and calculating the matching degree of the response text. The second comparison process can use exact matching algorithms, tolerance matching algorithms, or range matching algorithms, etc., and this embodiment of the invention does not impose any limitations.

[0074] S21422, Perform a second comparison process on the confirmation parameter configuration response information in the second evaluation result data information and the simulation intelligent agent configuration data information to obtain a second confirmation matching degree value.

[0075] It should be noted that the calculation method of the second confirmed matching degree value is the same as that of the second initial matching degree value. The confirmation parameter configuration response information is compared with the initial configuration content through the parameter extraction program and the numerical comparison program.

[0076] S21423, the second initial matching degree value and the second confirmed matching degree value are averaged to obtain the parameter configuration matching degree value.

[0077] It should be noted that the average calculation process refers to adding the second initial matching degree value and the second confirmed matching degree value and then dividing by 2 to obtain the parameter configuration matching degree value.

[0078] It is evident that the simulation agent capability evaluation method described in the embodiments of the present invention can quantify the accuracy and consistency of the simulation agent's memory of parameter configuration information.

[0079] In an optional embodiment, the step of calculating the logical configuration matching degree between the initial logical configuration response information in the initial evaluation result data information and the confirmation logical configuration response information in the second evaluation result data information to obtain a logical configuration matching degree value includes: S21431, constrain matching processing is performed on the initial logical configuration response information in the initial evaluation result data information to obtain a third initial matching degree value.

[0080] It should be noted that the constraint matching process refers to performing constraint condition matching on the response text of the simulated intelligent agent regarding the configuration constraints in a specific scenario, and detecting whether it conforms to the preset logical mapping configuration and boundary condition constraints. The constraint matching process can be performed using condition matching algorithms, logical reasoning algorithms, or pattern recognition algorithms, etc., and this embodiment of the invention does not limit the methods used.

[0081] S21432, perform constraint matching processing on the confirmation logic configuration response information in the second evaluation result data information to obtain the third confirmation matching degree value.

[0082] It should be noted that the calculation method of the third confirmation matching degree value is the same as that of the third initial matching degree value. The confirmation logic configuration response information is matched with the preset logic mapping configuration and boundary condition constraints through a pattern matching algorithm.

[0083] S21433, the third initial matching degree value and the third confirmed matching degree value are averaged to obtain the logical configuration matching degree value.

[0084] It should be noted that the average calculation process refers to adding the third initial matching degree value and the third confirmed matching degree value and then dividing by 2 to obtain the logical configuration matching degree value.

[0085] It is evident that the simulation agent capability evaluation method described in the embodiments of the present invention can quantify the accuracy and consistency of the simulation agent's memory of logical configuration information.

[0086] In another optional embodiment, the step of performing application capability evaluation processing on the configuration data information of the simulated intelligent agent and the evaluation scheme data information to obtain application capability assessment data information includes: S221, perform a first construction process on the configuration data information of the simulated intelligent agent and the evaluation scheme data information to obtain the preset scenario configuration data information.

[0087] It should be noted that the above-mentioned construction process refers to the configuration of the evaluation scheme data information in the system simulation system into a virtual simulation environment containing clearly defined response and handling rules and elements. The situational and environmental changes in the scenario are driven by the simulation engine according to the preset response and handling rules. The preset scenario configuration data information is stored in the scenario configuration database in a structured data format, including scenario element information (defining the entity objects and their attributes in the scenario), scenario response and handling rule information (defining the interaction logic and state transition logic between entities), and scenario initial situational information (defining the initial state of each entity at the start of the simulation).

[0088] It should be noted that for different types of simulated intelligent agents (such as scheduling simulated intelligent agents, detection and tracking simulated intelligent agents, communication simulated intelligent agents, interference simulated intelligent agents, output simulated intelligent agents, and response simulated intelligent agents), it is necessary to construct system simulation scenarios adapted to their instance types and functional configurations. The above scenario construction process can be handled using simulation scenario editing tools, scenario template configuration programs, or automated scenario generation algorithms, etc. Specifically, this embodiment of the invention does not impose limitations. The technology of the system simulation scenario supports the parallel operation and real-time interaction of multiple intelligent agent software instances in the same virtual environment.

[0089] S222, The simulation intelligent agent configuration data information and the preset scenario configuration data information are processed by simulation operation to obtain application evaluation simulation record data information.

[0090] It should be noted that the simulation operation process refers to placing the simulation agent software instance in the system simulation scenario based on the configuration data information of the simulation agent and the configuration data information of the preset scenario, and driving the simulation agent to run in the virtual environment through the simulation engine of the system simulation system. The simulation engine advances the simulation time in a discrete event-driven manner, sending environmental state data to the agent at each simulation time step. After receiving the data, the agent calls the inference module to generate output instructions and the execution module to output execution instructions. The application evaluation simulation record data information includes the input condition records, inference output records, execution instruction records, and execution result records of the simulation agent during the simulation process, which are stored in the simulation record database in a structured data format.

[0091] It should be noted that the system simulation system is a simulation application system based on parallel discrete event scheduling. The simulation operation and processing can be carried out using system simulation systems, distributed simulation platforms, or simulation engine software, etc. Specifically, the embodiments of the present invention do not limit the specific methods.

[0092] S223, perform output compliance detection processing on the application evaluation simulation record data information to obtain the output compliance detection value.

[0093] It should be noted that the output compliance detection processing refers to using a pattern matching algorithm to perform compliance detection on the inference output records of the simulated agent during the simulation process, determining whether each inference output conforms to a preset input-output mapping table. The output compliance detection value is the ratio of the number of outputs that conform to the mapping to the total number of outputs, and the calculation result is a floating-point number between 0 and 1. The output compliance detection processing can be performed using pattern matching algorithms, mapping table lookups, or large language model API detection, etc., and this embodiment of the invention is not limited to any particular method.

[0094] S224, Perform execution consistency detection processing on the application evaluation simulation record data information to obtain execution consistency detection value.

[0095] It should be noted that the execution consistency detection process refers to performing consistency checks on the execution records of the simulated agent during the simulation process using an instruction matching algorithm, analyzing the consistency between each execution result and its corresponding inference output instruction. The execution consistency detection value is the ratio of the number of execution results that match the inference output instructions to the total number of inference outputs, and the result is a floating-point number between 0 and 1. The execution consistency detection process can be performed using instruction matching algorithms, semantic similarity algorithms, pattern matching algorithms, or large language model API detection, etc., and this embodiment of the invention is not limited to any particular method.

[0096] S225, using the application capability comprehensive detection calculation model, calculate and process the output compliance detection value and the execution consistency detection value to obtain the application capability comprehensive detection value; The comprehensive application capability testing calculation model is as follows: ; ; ; In the formula, This is the comprehensive detection value of the application capability. The output compliance detection value is... The consistency check value is the value to be performed. , and These are the first application weight parameter, the second application weight parameter, and the third application weight parameter, respectively. To prevent small constants from being divided by zero.

[0097] It should be noted that the comprehensive application capability testing and calculation model is used to comprehensively quantify the joint operational performance of the inference module and execution module of the simulated intelligent agent. In the formula... This is an interaction term between the inference output and the execution result, used to detect the interface consistency between the inference module's output and the execution module's input: when both output consistency and execution consistency are high, the interaction term value is large, indicating correct data transfer and normal interface calls between the inference and execution modules; when either is low, the interaction term value decreases, indicating software integration problems such as data format mismatches or abnormal interface calls between modules. The computational model is executed through a numerical computation program, with two floating-point detection values ​​as input and one floating-point composite detection value as output.

[0098] It should be noted that the aforementioned weight parameters are stored in the configuration database and can be set by the user through the configuration interface, or automatically adjusted by the system based on the characteristics of the detection scene or historical detection data using a parameter optimization algorithm. Specifically, this embodiment of the invention does not impose limitations. For example, =0.5, =0.3, =0.2, =0.001.

[0099] S226, add the output compliance detection value, the execution consistency detection value, and the application capability comprehensive detection value to the application capability evaluation data information to obtain the application capability evaluation data information.

[0100] As can be seen, by implementing the simulation agent capability evaluation method described in the embodiments of the present invention, and by conducting parallel operation tests and pattern matching detection through a system simulation system, the operational performance and interface consistency of the simulation agent's inference module and execution module can be effectively detected.

[0101] In another optional embodiment, the step of performing emergent capability evaluation processing on the configuration data information of the simulated intelligent agent and the evaluation scheme data information to obtain emergent capability assessment data information includes: S231, perform a second construction process on the configuration data information of the simulated intelligent agent and the evaluation scheme data information to obtain the open scene configuration data information.

[0102] It should be noted that the above-mentioned construction process refers to the virtual simulation environment configured in the system simulation system without preset response rules for the evaluation scheme data information. The scenario only defines the basic network connection relationships between intelligent agents, without preset specific response rules and execution sequence scripts. The open scenario configuration data information is stored in the scenario configuration database in a structured data format, including scenario element information (defining the entity objects and their attributes in the scenario), initial connection information (defining the initial network topology between intelligent agents), and target state information (defining the target state of the simulation).

[0103] It should be noted that for different types of simulated intelligent agents (such as scheduling simulated intelligent agents, detection and tracking simulated intelligent agents, communication simulated intelligent agents, interference simulated intelligent agents, output simulated intelligent agents, response simulated intelligent agents, etc.), it is necessary to construct open scenarios adapted to their instance types and functional configurations. The simulation scenario without a pre-set response and processing rule system contains multiple simulated intelligent agent software instances. The scenario only specifies the target state and does not pre-set specific communication logic and execution sequences. It is used to test the self-organizing network performance of the simulated intelligent agent's communication module and the dynamic coordination performance of the data exchange module. The above scenario construction and processing can be carried out using simulation scenario editing tools, scenario template configuration programs, or automated scenario generation algorithms, etc. Specifically, this embodiment of the invention does not impose limitations.

[0104] S232, The simulation intelligent agent configuration data information and the open scene configuration data information are processed by simulation operation to obtain emergent evaluation simulation record data information.

[0105] It should be noted that the simulation operation process refers to placing the simulation agent software instance in a system simulation scenario without pre-set response rules, based on the configuration data information of the simulation agent and the configuration data information of the open scenario. The simulation agent is then driven to run in the virtual environment by the simulation engine of the system simulation system. The system simulation scenario without pre-set response rules refers to a virtual simulation environment configured in the system simulation system that does not contain explicit communication topology relationships and response rules. The situational and environmental changes in the scenario are dynamically determined by the simulation agent based on the output instructions of its own inference module. The simulation engine advances the simulation time in a discrete event-driven manner, sending environmental state data to the agent at each simulation time step. After receiving the data, the agent calls the inference module to generate output instructions and the execution module to output execution instructions. Before the emergent capability evaluation, the simulation agent has already run in a system simulation scenario with pre-set rules, retaining the initial rule configuration and historical running data. The emergent evaluation simulation record data information includes the simulation agent's inference output records, execution records, connection construction records, and information interaction records in the scenario without pre-set rules, stored in the simulation record database in a structured data format.

[0106] It should be noted that the above simulation operation process can be carried out using system simulation systems, distributed simulation platforms, or simulation engine software, etc., and the specific implementation of this invention is not limited thereto.

[0107] S233, perform open scene output detection processing on the emergence evaluation simulation record data information to obtain the open scene output compliance value.

[0108] It should be noted that the open-scene output detection processing refers to analyzing the inference output records of the simulated agent in an open scene (i.e., a simulated scene without preset rules) and detecting the output compliance of its inference module in a dynamic environment. The open-scene output compliance value is the ratio of the number of outputs that conform to the basic mapping logic to the total number of outputs, used to quantify the compliance of the outputs generated by the inference module in an open environment without clearly configured response rules. The above detection processing is implemented through a pattern matching algorithm, comparing the inference output records with the basic mapping logic and counting the number of outputs that conform to the mapping.

[0109] S234, perform connection construction detection processing on the emergence evaluation simulation record data information to obtain connection construction detection values.

[0110] It should be noted that the connection construction detection process refers to using graph theory algorithms to statistically analyze the changes in network connections between simulated agents in the system before and after the simulation, and to detect the self-organizing network performance of the communication modules of the simulated agents.

[0111] S235, perform information diffusion detection processing on the emergence evaluation simulation record data information to obtain information diffusion detection value.

[0112] It should be noted that the information diffusion detection and processing refers to using information propagation algorithms to statistically analyze the propagation path and coverage of important event messages in the agent network during the simulation process, and to detect the message routing efficiency and network transmission performance of the simulated agent communication module.

[0113] S236, calculate and process the open scene output compliance value, the connection construction detection value and the information diffusion detection value to obtain the comprehensive detection value of emergence capability; It should be noted that the above calculation and processing can be carried out using the emergence capability comprehensive detection calculation model, or it can be carried out using weighted summation algorithm, multi-factor fusion algorithm, comprehensive scoring algorithm or statistical analysis algorithm, etc. Specifically, the embodiments of the present invention are not limited.

[0114] The comprehensive detection and calculation model for emergence capability is as follows: ; ; ; In the formula, The comprehensive detection value of the emergence capability. Output a compliance value for the open scene. Detection values ​​are constructed for the connection. The information diffusion detection value is... and These are the first emergence weight parameters and the second emergence weight parameters, respectively. To prevent small constants from being divided by zero.

[0115] It should be noted that the emergence capability comprehensive detection and calculation model is used to comprehensively quantify the operational performance of the simulated intelligent agent's communication module and data exchange module under open scenario conditions. The formula adopts a multiplicative form. As a foundation, this reflects the holistic nature of multi-module collaborative operation: only when the inference module, communication module, and data exchange module all operate normally can the overall performance reach a high level; a failure in any module will lead to a significant drop in overall performance. (The interaction term in the formula...) and The interface coordination between the inference module and the communication module, and between the communication module and the data exchange module, is tested separately to detect data transmission anomalies and interface call failures between modules.

[0116] It should be noted that the aforementioned weight parameters are stored in the configuration database and can be set by the user through the configuration interface, or automatically adjusted by the system based on the characteristics of the detection scene or historical detection data using a parameter optimization algorithm. Specifically, this embodiment of the invention does not impose limitations. For example, =0.15, =0.15, =0.001.

[0117] S237, the open scene output compliance value, the connection construction detection value, the information diffusion detection value, and the emergence capability comprehensive detection value are added to the emergence capability evaluation data information to obtain the emergence capability evaluation data information.

[0118] As can be seen, by implementing the simulation agent capability evaluation method described in the embodiments of the present invention, stress testing and graph theory algorithm analysis under open scenario conditions can effectively detect the self-organizing network performance and message routing performance of the simulation agent communication module.

[0119] In an optional embodiment, the connection construction detection processing of the emergence evaluation simulation record data information to obtain connection construction detection values ​​includes: S2341, Perform pre-simulation connection statistical processing on the emergence evaluation simulation record data information to obtain the pre-simulation connection density value.

[0120] It should be noted that the pre-simulation connection statistics processing refers to using graph theory algorithms to count the number of unidirectional network connections between all simulated agents in the system before the simulation runs without pre-set response rules. For example, a unidirectional connection refers to a communication message sent by agent A to agent B, represented in the graph data structure as a directed edge from node A to node B. The statistics processing constructs a directed graph between agents by traversing the message records in the communication record database and calculates the number of edges in the directed graph.

[0121] S2342, Perform post-simulation connection statistics processing on the emergence evaluation simulation record data information to obtain the post-simulation connection density value.

[0122] It should be noted that the post-simulation connection statistics processing refers to using graph theory algorithms to count the number of unidirectional network connections between all simulated agents in the system after the simulation runs without pre-set response rules. The technical implementation of this statistical processing is the same as that of the pre-simulation connection statistics processing, which involves constructing a directed graph by traversing the communication record database after the simulation runs and calculating the number of edges.

[0123] S2343, The connection density values ​​before and after simulation are calculated and processed to obtain the connection construction detection values.

[0124] It should be noted that the above calculation and processing can be performed using a connection-based detection calculation model, or it can be performed using a ratio calculation algorithm, density change algorithm, graph theory analysis algorithm, or network topology analysis algorithm, etc. Specifically, the embodiments of the present invention are not limited.

[0125] The connection construction detection calculation model is as follows: ; ; In the formula, L Detection values ​​are constructed for the connection. The connection density value before the simulation. The simulated connection density value is... The formula for calculating connection density is as follows: Let be the number of one-way connections between all simulated agents in the system. This represents the number of simulated intelligent agents in the system. To prevent small constants from being divided by zero.

[0126] It should be noted that the connection construction and detection calculation model is used to quantify the self-organizing network performance of the simulated intelligent agent communication module. Connection density This represents the proportion of the actual number of network connections established in the system to the maximum possible number of connections. The result is a floating-point number between 0 and 1, and in graph theory, it is called the edge density of a directed graph. (Connection establishment detection value) L The ratio of the connection density after simulation to the connection density before simulation is given by... This indicates that the network connection has been enhanced during the simulation, and the self-organizing network function of the communication module is normal; when This indicates a weakened network connection, which may be due to a communication module malfunction or network congestion.

[0127] For example, =0.001, used to prevent division by zero anomalies.

[0128] As can be seen, by implementing the simulated intelligent agent capability evaluation method described in the embodiments of the present invention, the edge density change of the network topology can be calculated using graph theory algorithms, which can quantify and detect the self-organizing network performance and network connection stability of the simulated intelligent agent communication module.

[0129] In another optional embodiment, the step of performing information diffusion detection processing on the emergence evaluation simulation record data to obtain information diffusion detection values ​​includes: S2351, perform important event identification processing on the emergence evaluation simulation record data information to obtain an important event data information set.

[0130] It should be noted that the important event identification process refers to using a pattern matching algorithm to filter out event records that have a significant impact on the simulation situation from the simulation record database. The important event data information set includes several important event data information, each of which is stored in a structured data format, including an event identifier field (a unique identifier in UUID format), an event content field (an event description in text format), and an event occurrence time field (timestamp format).

[0131] S2352, default s=1.

[0132] S2353, Perform reception statistical processing on the s-th important event data information in the important event data information set to obtain the number of receiving agents for the s-th important event data information.

[0133] It should be noted that the aforementioned reception statistical processing refers to querying the database to determine the number of simulated agents that received the s-th important event message, i.e., the number of simulated agents to which the event message reached. The technical implementation of this reception statistical processing includes: searching the data storage tables of each agent using a database query statement for records containing the identifier of the s-th important event, and counting the number of agents returning the returned records.

[0134] S2354, determine whether s is equal to S, and obtain a first determination result, where S is the number of important event data information in the important event data information set; When the first judgment result is negative, increment s by 1 and execute S2353; If the first judgment result is yes, execute S2355.

[0135] S2355, calculate and process the number of receiving agents corresponding to all the important event data information to obtain the information diffusion detection value.

[0136] It should be noted that the above calculation and processing can be performed using an information diffusion detection calculation model, or by using an average value calculation algorithm, coverage statistics algorithm, propagation efficiency analysis algorithm, or network propagation analysis algorithm, etc. Specifically, the embodiments of the present invention are not limited.

[0137] The information diffusion detection calculation model is as follows: ; In the formula, S The information diffusion detection value is... For the first i The number of receiving agents corresponding to each important event data information. IThe total number of important event data information in the set of important event data information. n This represents the number of simulated intelligent agents in the system. .

[0138] It should be noted that the information diffusion detection calculation model is used to quantify the message routing efficiency and network transmission performance of the simulated intelligent agent communication module. Information diffusion detection value. S This represents the average proportion of simulated agents whose messages for each important event reach them, calculated as a floating-point number between 0 and 1. When S When the value is close to 1, it indicates that the message routing function of the communication module is normal, and messages can propagate fully in the agent network; when... S A lower value indicates a fault in the communication module, such as message loss, routing errors, or network partitioning, requiring technical troubleshooting.

[0139] As can be seen, by implementing the simulation agent capability evaluation method described in the embodiments of the present invention, the message routing efficiency and network transmission reliability of the simulation agent communication module can be quantitatively detected through message propagation tracking and coverage statistics.

[0140] In another optional embodiment, the step of comprehensively calculating and processing the capability assessment data to obtain the target capability level classification result information includes: S31, Using the comprehensive capability detection calculation model, the capability assessment data information is calculated and processed to obtain the comprehensive capability detection value; The comprehensive capability detection calculation model is as follows: ; ; ; In the formula, The comprehensive capability detection value is... This refers to the comprehensive detection value of basic capabilities within the aforementioned basic capability assessment data. This refers to the comprehensive detection value of application capabilities within the application capability assessment data. The comprehensive detection value of emergent capability in the emergent capability assessment data information. , , and These are the first comprehensive weight parameter, the second comprehensive weight parameter, the third comprehensive weight parameter, and the fourth comprehensive weight parameter, respectively. To prevent small constants from being divided by zero.

[0141] It should be noted that the comprehensive capability detection and calculation model is used to comprehensively quantify the overall operational performance of the simulated intelligent agent software system. In the formula... The weighted sum of the detection data from the three dimensions reflects the performance contribution of the memory module, the inference module, and the communication module, respectively. This is an interactive item for 3D detection data, used to detect the interface coordination and system integration quality between various software modules, reflecting the integrity and module balance of the software system.

[0142] It should be noted that the above weight parameters are stored in the configuration database and can be flexibly set through the configuration interface according to the instance type of the simulated agent and the application scenario. For scenarios that emphasize the stability detection of the memory module, the weight can be increased. For scenarios that emphasize testing the response capability of the inference module, the capacity can be increased. For scenarios that emphasize testing the networking capabilities of communication modules, the capacity can be increased. For example, , , , , .

[0143] S32, the comprehensive capability detection value and the preset capability level classification threshold data information are processed to obtain capability level information.

[0144] It should be noted that the above capability level classification threshold data is stored in the configuration database in a structured data format, including the classification thresholds and level descriptions for the six capability levels: Level 1 (High Capability): Comprehensive capability test value of 90-100, balanced performance of each software module, high stability of memory module, fast response of inference module, and strong networking performance of communication module, suitable for complex multi-agent communication simulation scenarios; Level 2 (Excellent): Comprehensive capability test value 80-89, main software modules have excellent performance, can efficiently complete complex simulation tasks, and are suitable for conventional multi-agent simulation scenarios; Level 3 (Compliance Type): Comprehensive capability test value 70-79, the overall operating performance of the software system meets the standards, it can effectively cope with the simulation scenarios with preset response and processing rules, has a certain degree of open scenario adaptability, and is suitable for special simulation tasks; Level 4 (Basic): Comprehensive capability test value 60-69, memory module performance meets the standard, inference module or communication module has performance shortcomings, suitable for simple simulation scenarios; Level 5 (Under Optimization): Overall capability test value 50-59, some software modules do not meet the performance standards, and software optimization and fault repair are required; Level 6 (Fault-related): The comprehensive capability test value is less than 50, indicating that the software system has serious performance problems or faults and the core software modules need to be rebuilt.

[0145] The above processing is achieved through an interval judgment algorithm, which compares the comprehensive capability detection value with the range of comprehensive capability detection values ​​corresponding to each level in the preset capability level classification threshold data information to determine the level. The capability level information is stored in a structured data format, including a level number field (integer type, value range 1-6), a level name field (text type, such as "high capability type", "excellent type", etc.), and a level description field (text type, containing the performance characteristics and applicable scenarios of the level). For example, when the comprehensive capability detection value is 85, the capability level information is: level number = 2, level name = "excellent type", level description = "The main software modules have excellent operating performance, can efficiently complete complex simulation tasks, and are suitable for conventional multi-agent simulation scenarios".

[0146] S33, add the comprehensive capability detection value and the capability level information to the target capability level classification result information to obtain the target capability level classification result information.

[0147] It is evident that the simulation agent capability evaluation method described in the embodiments of the present invention, through weighted calculation and interval classification of multi-dimensional detection data, can provide a quantitative basis for performance optimization and fault diagnosis of simulation agent software systems.

[0148] Example 2 Please see Figure 2 , Figure 2 This is a schematic diagram of the structure of a capability evaluation device for a large-model-based simulated intelligent agent, as disclosed in an embodiment of the present invention. Figure 2 The described simulated intelligent agent capability assessment device is applied in a simulation assessment system, such as a local server or cloud server used for game deduction and simulation assessment, etc., and the embodiments of the present invention are not limited thereto. Figure 2 As shown, the simulated intelligent agent capability evaluation device includes: The acquisition module 201 is used to acquire configuration data information of the simulated intelligent agent and evaluation scheme data information. The capability assessment module 202 is used to process the configuration data information of the simulated intelligent agent and the assessment scheme data information to obtain capability assessment data information; the capability assessment data information includes basic capability assessment data information, application capability assessment data information and emergent capability assessment data information. The comprehensive calculation module 203 is used to perform comprehensive calculation and processing on the capability assessment data information to obtain the target capability level classification result information.

[0149] As can be seen, the simulated intelligent agent capability evaluation device described in the embodiments of the present invention can comprehensively test the operating performance of the memory module, inference module, execution module and communication module of the simulated intelligent agent software system by calling the large language model API interface, system simulation system, semantic similarity calculation algorithm, pattern matching algorithm and graph theory algorithm.

[0150] Example 3 Please see Figure 3 , Figure 3 This is a schematic diagram of another capability evaluation device for simulated intelligent agents based on large models, as disclosed in an embodiment of the present invention. Figure 3 The described simulated intelligent agent capability assessment device is applied in a simulation assessment system, such as a local server or cloud server used for system confrontation simulation and assessment, etc., and the embodiments of the present invention are not limited thereto. Figure 3 As shown, the simulated intelligent agent capability evaluation device includes: Processor 301; A memory 302 containing executable program code is coupled to the processor 301; The processor 301 calls the executable program code stored in the memory 302 to execute some or all of the steps of the capability assessment method for a large-model-based simulated intelligent agent in Embodiment 1.

[0151] It should be noted that the processor 301 can be a central processing unit (CPU), a graphics processing unit (GPU), or an application-specific integrated circuit (ASIC) for executing computational instructions in the program code. The memory 302 can be random access memory (RAM), read-only memory (ROM), or a solid-state drive (SSD) for storing program code and runtime data. The processor 301 and the memory 302 transmit data via a system bus.

[0152] As can be seen, the simulation agent capability evaluation device described in the embodiments of the present invention can comprehensively test the running performance of the simulation agent software system by executing program code in the memory through the processor, calling large language model API interface, system simulation system, semantic similarity calculation algorithm, pattern matching algorithm and graph theory algorithm and other technical means.

[0153] Example 4 This invention discloses a computer-readable storage medium storing computer instructions. When the computer instructions are invoked, they are used to execute some or all of the steps of the capability assessment method for a large-model-based simulated intelligent agent according to Embodiment 1.

[0154] Example 5 This invention discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps of the capability assessment method for a large-model-based simulated intelligent agent according to Embodiment 1.

[0155] The system embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0156] Through the detailed description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electronically erasable rewritable read-only memory (EEPROM), read-only optical disc (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0157] Finally, it should be noted that the capability evaluation method and apparatus for large-model-based simulated intelligent agents disclosed in the embodiments of the present invention are merely preferred embodiments of the present invention and are only used to illustrate the technical solutions of the present invention, not to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for evaluating the capabilities of simulated intelligent agents based on large models, characterized in that, The method includes: S1, acquire configuration data information and evaluation scheme data information of the simulated intelligent agent; S2, process the configuration data information of the simulated intelligent agent and the evaluation scheme data information to obtain capability assessment data information; the capability assessment data information includes basic capability assessment data information, application capability assessment data information and emergent capability assessment data information; S3, perform comprehensive calculation and processing on the capability assessment data information to obtain the target capability level classification result information; S2 includes: S21, perform basic capability evaluation processing on the configuration data information of the simulated intelligent agent and the evaluation scheme data information to obtain basic capability evaluation data information; S22, Perform application capability evaluation processing on the configuration data information of the simulated intelligent agent and the evaluation scheme data information to obtain application capability evaluation data information; S23, perform emergent capability evaluation processing on the configuration data information of the simulated intelligent agent and the evaluation scheme data information to obtain emergent capability evaluation data information, including: S231, perform a second construction process on the configuration data information of the simulated intelligent agent and the evaluation scheme data information to obtain open scene configuration data information; S232, The simulation intelligent agent configuration data information and the open scene configuration data information are processed by simulation operation to obtain emergent evaluation simulation record data information; S233, perform open scene output detection processing on the emergence evaluation simulation record data information to obtain the open scene output compliance value; S234, Perform connection construction detection processing on the emergence evaluation simulation record data information to obtain connection construction detection values; S235, perform information diffusion detection processing on the emergence evaluation simulation record data information to obtain information diffusion detection value; S236, using the comprehensive detection and calculation model of emergence capability, the open scene output compliance value, the connection construction detection value and the information diffusion detection value are calculated and processed to obtain the comprehensive detection value of emergence capability; The comprehensive detection and calculation model for emergence capability is as follows: ; ; ; In the formula, The comprehensive detection value of the emergence capability. The open scene outputs a compliance value, representing the ratio of the number of outputs conforming to the basic mapping logic to the total number of outputs. This value characterizes the inference output records of the simulated agent inference module within the open scene and detects the output compliance of the simulated agent inference module in dynamic environments. A detection value is constructed for the connection to characterize the ratio of the connection density after simulation to the connection density before simulation, where the connection density represents the proportion of the actual number of network connections established to the maximum possible number of connections. The information diffusion detection value represents the average proportion of simulated agents reached by each important event message, indicating the propagation coverage of important event messages in the agent network. and These are the first emergence weight parameters and the second emergence weight parameters, respectively. To prevent division by zero of small constants; S237, the open scene output compliance value, the connection construction detection value, the information diffusion detection value, and the emergence capability comprehensive detection value are added to the emergence capability evaluation data information to obtain the emergence capability evaluation data information.

2. The capability evaluation method for simulated intelligent agents based on large models according to claim 1, characterized in that, The basic capability assessment processing of the configuration data information of the simulated intelligent agent and the evaluation scheme data information to obtain basic capability evaluation data information includes: S211, perform initial question-and-answer evaluation processing on the configuration data information of the simulated intelligent agent and the evaluation scheme data information to obtain initial evaluation result data information; S212, Process the configuration data information of the simulated intelligent agent to obtain simulation operation record data information; S213, perform a second evaluation process on the configuration data information of the simulated intelligent agent and the evaluation scheme data information to obtain the second evaluation result data information; S214, perform matching degree calculation on the initial evaluation result data information, the second evaluation result data information, and the simulation intelligent agent configuration data information to obtain basic capability evaluation data information.

3. The capability evaluation method for simulated intelligent agents based on large models according to claim 2, characterized in that, The initial question-and-answer evaluation process is performed on the configuration data information of the simulated intelligent agent and the evaluation scheme data information to obtain initial evaluation result data information, including: S2111, preprocess the evaluation information set in the evaluation scheme data information to obtain a preprocessed evaluation information set; S2112, The preprocessed evaluation information set and the simulated intelligent agent configuration data information are processed to obtain evaluation prompt word data information; S2113, using the evaluation prompt word data information and the large language model, the simulation agent corresponding to the simulation agent configuration data information is processed to obtain the original response data information; S2114, The original response data information is processed by response parsing to obtain the initial evaluation result data information.

4. The capability assessment method for simulated intelligent agents based on large models according to claim 1, characterized in that, The process of performing application capability evaluation on the configuration data information of the simulated intelligent agent and the evaluation scheme data information to obtain application capability assessment data information includes: S221, Perform a first construction process on the configuration data information of the simulated intelligent agent and the evaluation scheme data information to obtain preset scenario configuration data information; S222, The simulation operation processing is performed on the configuration data information of the simulated intelligent agent and the configuration data information of the preset scenario to obtain application evaluation simulation record data information; S223, Perform output compliance detection processing on the application evaluation simulation record data information to obtain the output compliance detection value; S224, Perform execution consistency detection processing on the application evaluation simulation record data information to obtain the execution consistency detection value; S225, using the application capability comprehensive detection calculation model, calculate and process the output compliance detection value and the execution consistency detection value to obtain the application capability comprehensive detection value; The comprehensive application capability testing calculation model is as follows: ; ; ; In the formula, P This is the comprehensive detection value of the application capability. The output conformity detection value characterizes whether the inference output of the simulated agent during the simulation process conforms to a preset input-output mapping table, and represents the ratio of the number of inference outputs that conform to the input-output mapping table to the total number of inference outputs. The execution consistency detection value represents the ratio of the number of execution results that match the inference output instructions to the total number of inference outputs, characterizing the consistency between the execution results and the inference output instructions of the simulated agent during the simulation process. , and These are the first application weight parameter, the second application weight parameter, and the third application weight parameter, respectively. To prevent division by zero of small constants; S226, add the output compliance detection value, the execution consistency detection value, and the application capability comprehensive detection value to the application capability evaluation data information to obtain the application capability evaluation data information.

5. The capability evaluation method for simulated intelligent agents based on large models according to claim 1, characterized in that, The process of comprehensively calculating and processing the capability assessment data to obtain the target capability level classification result information includes: S31, Using the comprehensive capability detection calculation model, the capability assessment data information is calculated and processed to obtain the comprehensive capability detection value; The comprehensive capability detection calculation model is as follows: ; ; ; In the formula, The comprehensive capability detection value is... This refers to the comprehensive detection value of basic capabilities within the aforementioned basic capability assessment data. This refers to the comprehensive detection value of application capabilities within the application capability assessment data. The comprehensive detection value of emergent capability in the emergent capability assessment data information. , , and These are the first comprehensive weight parameter, the second comprehensive weight parameter, the third comprehensive weight parameter, and the fourth comprehensive weight parameter, respectively. To prevent division by zero of small constants; S32, Process the comprehensive capability detection value and the preset capability level classification threshold data information to obtain capability level information; S33, add the comprehensive capability detection value and the capability level information to the target capability level classification result information to obtain the target capability level classification result information.

6. A capability evaluation device for simulated intelligent agents based on large models, characterized in that, The device includes: processor; A memory coupled to the processor stores executable program code; The processor calls the executable program code stored in the memory to execute the capability evaluation method for a large-model-based simulated intelligent agent as described in any one of claims 1-5.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions, which, when invoked, are used to execute the capability assessment method for a large-model-based simulated intelligent agent as described in any one of claims 1-5.

Images