Model testing method and device, electronic device and storage medium
By accessing the target testing environment and constructing test requests using the model context protocol, the efficiency and accuracy issues of large language model testing are solved, achieving efficient automated testing.
Patent Information
- Application Number
- CN202511893361.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-15
- Publication Date
- 2026-03-06
AI Technical Summary
Traditional automated testing methods cannot efficiently handle the high dynamism of large language models and the need for large-scale corpora, resulting in low efficiency and high cost of manual verification.
By accessing the target testing environment, using the model context protocol to obtain test cases and environment context information, constructing target test requests, and determining test results based on the model response results, automated testing is achieved.
It improves the efficiency and accuracy of testing large language models, reduces human intervention, and supports automated testing of large-scale corpora.
Smart Images
Figure CN121614403A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of automated testing technology, and in particular to a model testing method, apparatus, electronic device, and storage medium. Background Technology
[0002] With the development of computer technology and the increasing volume of data, more and more large language models are being applied to various scenarios. In the development and deployment of large language model applications, testing is a crucial step in ensuring the quality of model output. With the widespread application of large language models in scenarios such as intelligent customer service, content generation, and code assistance, testing demands are exploding. In these scenarios, the model needs to generate corresponding outputs from various user inputs (corpora), and testers need to verify whether the outputs meet expectations. Testers need to prepare tens of thousands of test corpora and verify the correctness of the model's returned results one by one. This traditional testing method is inefficient, requires a large amount of manual input, and still needs to repeat manual verification after model iterations and updates, increasing labor and time costs. Traditional automated testing methods rely on predefined test scripts and cannot adapt to the high dynamism and large-scale corpus requirements of large language model testing. Therefore, there is an urgent need for an automated testing method for large language model application scenarios that can efficiently handle large-scale corpus testing needs, reduce manual intervention, and improve testing efficiency and accuracy.
[0003] Currently, no effective solution has been proposed for how to efficiently and automatically perform testing on large language models in related technologies. Summary of the Invention
[0004] This application provides a model testing method, apparatus, electronic device, and storage medium to at least address the problem in the related art of how to efficiently and automatically implement testing for large language models.
[0005] Firstly, embodiments of this application provide a model testing method.
[0006] In some embodiments, the model testing method includes:
[0007] Access the target testing environment, which is configured to be compatible with the Model Context Protocol;
[0008] The target test case and the environment context information corresponding to the target test environment are obtained through the model context protocol, and a target test request is constructed based on the target test case and the environment context information;
[0009] The target test request is sent to the target large language model, so that the target large language model outputs the model response result according to the target test request;
[0010] Determine the target expected output in the target test case, and determine the target test result based on the model response result and the target expected output.
[0011] In some embodiments, the access target testing environment, configured to be suitable for the Model Context Protocol, includes:
[0012] Obtain the model API information, database information, and UI information of the target testing environment;
[0013] Based on the model API information, the database information, and the UI information, the target test environment is accessed via the model context protocol, and the target test environment is configured to be compatible with the model context protocol.
[0014] In some embodiments, obtaining the target test case and the environment context information corresponding to the target test environment through the model context protocol, and constructing the target test request based on the target test case and the environment context information, includes:
[0015] The model API status information, database snapshot information, and UI structure synchronization information corresponding to the target test environment are obtained through the model context protocol.
[0016] Based on the model API status information, the database snapshot information, and the UI structure synchronization information, construct the target test request according to the protocol corresponding to the model API.
[0017] In some embodiments, obtaining the model API status information, database snapshot information, and UI structure synchronization information corresponding to the target test environment through the model context protocol includes:
[0018] A health check request is sent to the model API via the model context protocol to obtain the model API status information output by the model API in response to the health check request.
[0019] The database information is queried using the model context protocol to determine database snapshot information including test case data;
[0020] The model context protocol is used to obtain UI structure synchronization information, including the current page DOM structure and UI snapshot.
[0021] In some embodiments, the model testing method further includes:
[0022] Acquire test data and parse the test data to determine the input prompt information, expected output information, test condition information and test priority information included in the test data;
[0023] Based on the input prompts, expected outputs, test conditions, test priority, and preset test templates, generate multiple structured test cases, including the target test cases.
[0024] In some embodiments, determining the target expected output in the target test case, and determining the target test result based on the model response result and the target expected output, includes:
[0025] When the target expected output is fixed text, the model response result and the target expected output are compared as strings to determine the target test result;
[0026] When the expected target output is a fuzzy rule, the target similarity between the model response result and the expected target output is calculated, and the target test result is determined based on the target similarity.
[0027] If the expected output of the target is executable code, the executable code is dynamically run to obtain the anomaly judgment result, and the target test result is determined based on the anomaly judgment result.
[0028] In some embodiments, after determining the target expected output in the target test case and determining the target test result based on the model response result and the target expected output, the method further includes:
[0029] The target test results are analyzed to determine test result report information, including test pass rate information, test failure rate information, and test failure reason information;
[0030] Based on the test result report information, adjust the corresponding model training data and model training parameters during the training process of the target large language model.
[0031] Secondly, embodiments of this application provide a model testing device.
[0032] In some embodiments, the model testing device includes a test environment access module, a test request construction module, a response result output module, and a test result determination module.
[0033] The test environment access module is used to access the target test environment, which is configured to be compatible with the model context protocol.
[0034] The test request construction module is used to obtain the target test case and the environment context information corresponding to the target test environment through the model context protocol, and construct the target test request according to the target test case and the environment context information;
[0035] The response result output module is used to send the target test request to the target large language model, so that the target large language model outputs the model response result according to the target test request;
[0036] The test result determination module is used to determine the target expected output in the target test case, and to determine the target test result based on the model response result and the target expected output.
[0037] Thirdly, embodiments of this application provide an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the model testing method as described in the first aspect above.
[0038] Fourthly, embodiments of this application provide a storage medium storing a computer program thereon, which, when executed by a processor, implements the model testing method as described in the first aspect above.
[0039] Compared to related technologies, the model testing method, apparatus, electronic device, and storage medium provided in this application access a target testing environment configured to be compatible with the Model Context Protocol (MTP). They obtain target test cases and corresponding environment context information through the MTP, construct a target test request based on the target test cases and environment context information, send the target test request to the target large language model, and enable the target large language model to output model response results based on the target test request. They determine the target expected output in the target test cases and, based on the model response results and the target expected output, determine the target test result. This solves the problem of how to efficiently and automatically implement testing for large language models, efficiently handles large-scale corpus testing needs in large model application scenarios, reduces manual intervention, and improves testing efficiency and accuracy.
[0040] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description
[0041] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:
[0042] Figure 1 This is a hardware structure block diagram of the terminal of the model testing method according to an embodiment of this application;
[0043] Figure 2 This is a flowchart of a model testing method according to an embodiment of this application;
[0044] Figure 3 This is a flowchart of another model testing method according to an embodiment of this application;
[0045] Figure 4 This is a flowchart of a model testing method according to a preferred embodiment of this application;
[0046] Figure 5 This is a structural block diagram of a model testing apparatus according to an embodiment of this application. Detailed Implementation
[0047] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application. Furthermore, it is understood that although the efforts made in such a development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, modifications to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.
[0048] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.
[0049] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application means two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The terms “first,” “second,” “third,” etc., used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.
[0050] The method embodiments provided in this example can be executed on a terminal, computer, or similar computing device. Taking running on a terminal as an example, Figure 1 This is a hardware structure block diagram of the terminal of the model testing method according to an embodiment of the present invention. For example... Figure 1 As shown, a terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. Optionally, the terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the terminal described above. For example, the terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.
[0051] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the model testing method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0052] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.
[0053] This embodiment provides a model testing method. Figure 2 This is a flowchart of a model testing method according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps:
[0054] Step S201: Access the target test environment, which is configured to be compatible with the model context protocol.
[0055] In this embodiment, a Model Context Protocol Server (MCP Server) is configured to connect the large language model and the testing environment, thereby implementing the model testing method. Specifically, the MCP Server is installed and configured (i.e., the MCP Server is initialized) to connect to the target testing environment, which is configured to be compatible with the Model Context Protocol. Optionally, the environment preparation for the target testing environment includes: ensuring that the server meets hardware requirements (such as the number of CPU cores, memory size, GPU support, etc.) and installing necessary dependent libraries (such as Docker, Python 3.8+, Node.js, etc.); deploying the MCP Server through an installation package or source code compilation, supporting Linux / Windows environments; configuring network firewall rules to open ports for communication between the MCP Server and the testing environment (such as HTTP / HTTPS ports, gRPC ports). Optionally, the implementation and parameter settings for connecting the MCP Server to the test environment can be as follows: Fill in the connection information for the test environment, including the model API address (such as a RESTful API endpoint or gRPC service address); the database connection string, including username, password, and table name (such as jdbc:mysql: / / host:port / db for MySQL); and the UI test target URL (such as the entry address of a web application or the debugging port for a mobile device). Furthermore, this embodiment supports multi-environment configuration (development / testing / production), which can be switched via configuration files or environment variables. The installed and configured MCP Server is then connected to the target test environment via the Model Context Protocol.
[0056] Step S202: Obtain the target test case and the environment context information corresponding to the target test environment through the model context protocol, and construct the target test request based on the target test case and the environment context information.
[0057] In this embodiment, the MCP Server obtains the environmental context information of the target test environment in real time through the model context. This environmental context information may include the model API status, UI structure, etc. Further, the MCP Server constructs a target test request corresponding to the target test case according to the model API protocol, ensuring that the target test request includes input prompts (such as "query xxx's personnel file" in the example below) and test conditions (such as max_tokens=50, temperature=0.7 in the example below). In this embodiment, taking the OpenAI API protocol as an example, the following target test request is constructed:
[0058]
[0059] In addition, if the test case contains special conditions (such as prohibiting the output of specific content), you can add stop_sequences or content_filter to the request.
[0060] Step S203: Send the target test request to the target large language model so that the target large language model outputs the model response result according to the target test request.
[0061] In this embodiment, the MCP Server sends a target test request to the target large language model and receives a response. Specifically, the MCP Server can send a request to the model API of the target large language model via HTTP / gRPC. For example, the model returns the following response:
[0062]
[0063] It is worth mentioning that, based on the model response results, the MCP Server can mark the request status as SUCCESS / TIMEOUT / ERROR. For example, if the request times out (e.g., no response for more than 5 seconds), the MCP Server marks the request status as TIMEOUT and records it.
[0064] In this embodiment of the application, multimodal testing support can also be implemented, that is, in addition to text input, image and audio input (such as transmission via Base64 encoding) can be supported; multi-field responses of model output (such as {"text": "Hello", "confidence": 0.95}) can be captured; and asynchronous test processing can be supported, that is, for long-time tasks (such as model training), polling results (such as querying / job / status every 30 seconds) can be supported, and an asynchronous timeout time (such as 1 hour) can be set, and the result can be marked as "incomplete" after the timeout.
[0065] Step S204: Determine the target expected output in the target test case, and determine the target test result based on the model response result and the target expected output.
[0066] Furthermore, the target expected output in the target test case is determined, such as whether the name is output correctly. The model response result is matched with the target expected output, and the target test result is determined based on the matching result. The intelligent matching rules in this application embodiment include complete matching, partial matching, and custom validators. Among them, complete matching includes: strict mode - the strings must be completely identical (including punctuation); lenient mode - ignore capitalization and extra spaces. Partial matching includes: using similarity algorithms (e.g., cosine similarity ≥ 0.8 is considered partially passed); supporting keyword matching (e.g., if the expected output contains "success", it passes). Custom validators include: writing Python scripts to implement complex logic (e.g., validating the nesting relationship of JSON fields).
[0067] Through the above steps, this embodiment configures an MCP Server and connects to a target test environment suitable for the Model Context Protocol (MCP). It obtains the target test cases and the corresponding environment context information of the target test environment through the MCP, constructs a target test request based on this, and sends the target test request to the target large language model. The target large language model then outputs a model response result based on the target test request. Based on the model response result and the expected output in the target test cases, the target test result is determined. This efficiently processes large-scale test corpora, achieving automated testing in large language model application scenarios. It solves the problem of how to efficiently and automatically implement testing for large language models, enabling real-time perception of the test environment, immediate feedback of test results, reduced manual intervention, and improved testing efficiency and accuracy.
[0068] In some embodiments, step S201 includes:
[0069] Step S2011: Obtain the model API information, database information, and UI information of the target test environment.
[0070] In this embodiment of the application, the MCP Server obtains the key resources of the target test environment, including model API information, database information, and UI information.
[0071] Step S2012: Based on the model API information, database information, and UI information, access the target test environment through the model context protocol. The target test environment is configured to be compatible with the model context protocol.
[0072] Furthermore, the MCP Server accesses key resources of the target testing environment through a standardized model context protocol. Specifically: It determines the model API based on model API information to obtain the calling interface for large language models, supports dynamic model version switching (e.g., specified via the `model_version=1.2` parameter), and configures request timeouts (e.g., default 5 seconds), retry mechanisms (e.g., 3 retries), and rate limiting strategies (e.g., QPS limits); it obtains test data and historical test results based on database information, and defines SQL query templates or ORM mappings, such as obtaining high-priority test cases via `SELECT * FROM test_cases WHEREpriority='HIGH'`, and supports transaction rollback to ensure test data consistency; and it obtains the front-end page structure based on UI information for UI testing.
[0073] In addition, when connecting the MCP Server to the target testing environment, it is necessary to configure the MCP Server parameters, such as model version, test scope, and timeout settings. Model test scope settings include: dividing by functional module (e.g., testing only "text generation" or "dialogue system"); supporting regular expression filtering of input (e.g., testing only corpora containing the [urgent] tag). Timeout settings include: setting an API call timeout threshold (e.g., 10 seconds), automatically marking it as "test timeout" after a timeout. Furthermore, configure the MCPInspector tool to monitor and visualize connection status in real time (e.g., database heartbeat detection, API latency heatmap), and provide log retrieval functionality (e.g., filtering ERROR-level logs). Also, configure debug modes, such as: supporting manual triggering of individual test cases and real-time viewing of request / response data; providing a Mock function to simulate the model API returning preset results.
[0074] Through the above steps, this embodiment of the application specifically accesses the target test environment through the model context protocol based on model API information, database information, and UI information, thereby achieving real-time perception of the test environment and immediate feedback of test results. It supports continuous testing during the development phase, discovers defects earlier than traditional testing, and improves testing efficiency and stability.
[0075] In some embodiments, step S202 includes:
[0076] Step S2021: Obtain the model API status information, database snapshot information, and UI structure synchronization information corresponding to the target test environment through the model context protocol.
[0077] In this embodiment, the environment context information corresponding to the target test environment may include model API status, UI structure, etc., and more specifically, may include model API status information, database snapshot information, and UI structure synchronization information. Model API status information is used to obtain model load information (such as GPU utilization and queued task count) through a health check interface (e.g., / health); and to dynamically adjust test concurrency (e.g., automatically reducing QPS based on API response time). Database snapshot information is used to back up test data before execution and automatically roll back in case of failure (e.g., via CREATE TABLE backup_xxx AS SELECT * FROM xxx). UI structure synchronization information is used to compare the DOM hash value of the previous test, identify UI changes, and trigger alarms.
[0078] Step S2022: Based on the model API status information, database snapshot information, and UI structure synchronization information, construct the target test request according to the protocol corresponding to the model API.
[0079] Furthermore, based on the model API status information, database snapshot information, and UI structure synchronization information, and in the target test environment ready state, a target test request is constructed according to the protocol corresponding to the model API, based on the target test case data (input, expected output, test conditions, etc.).
[0080] Through the above steps, this application provides a specific method for constructing target test requests based on target test case data, realizing natural language-driven testing without the need to write complex test scripts, and further improving the degree of test automation and the accuracy of results.
[0081] In some embodiments, step S2021 includes:
[0082] Step S2121: Send a health check request to the model API through the model context protocol to obtain the model API status information output by the model API in response to the health check request.
[0083] In this embodiment of the application, performing an API status check to determine the model API status information includes: sending a health check request (such as GET / health) to the model API through the model context protocol; the model API returning a status (such as {"status": "healthy", "version": "1.2.3"}); if the API is unavailable, the MCP Server records the error, suspends the test, and notifies the test administrator.
[0084] Step S2221: Query database information through the model context protocol to determine database snapshot information including test case data.
[0085] In this embodiment of the application, the acquisition of database snapshot information and data preparation includes: obtaining test case data by querying the database (e.g., SELECT * FROM test_cases WHERE priority = 'HIGH'); optionally, a temporary data table can also be created to store the data state before the test (for rollback).
[0086] Step S2321: Obtain UI structure synchronization information, including the current page DOM structure and UI snapshot, through the model context protocol.
[0087] In this embodiment of the application, the acquisition of UI structure synchronization information includes: if the test involves UI (such as web automation testing), the MCP Server obtains the current page DOM structure through Selenium / Playwright; and stores UI snapshots (such as HTML / CSS / screenshots) for subsequent comparison.
[0088] Through the above steps, this application provides a method for obtaining model API status information, database snapshot information, and UI structure synchronization information through the model context protocol, thereby improving the feasibility of automated testing and the accuracy of test results.
[0089] This application also provides a model testing method. Figure 3 This is a flowchart of another model testing method according to an embodiment of this application, such as... Figure 3 As shown, based on steps S201 to S204, the process further includes the following steps:
[0090] Step S301: Obtain test data and parse the test data to determine the input prompt information, expected output information, test condition information and test priority information included in the test data.
[0091] The test corpus is acquired and parsed using NLP technology to extract key information, including: input prompt information - prompts input by the user, expected output information - the expected model output, test condition information - special test conditions (such as input length limits, specific keywords), and test priority information. In addition, embodiments of this application can also combine model-based testing (MBT) technology to generate test cases from the model behavior state machine, or search-based testing (SBT) technology can be used to optimize the generation of test cases.
[0092] In this embodiment, users can input test requirements, including test corpora and expected results, through a natural language interface (such as a web interface or command-line tool), or by uploading test corpora in batches via an API interface, or by automatically importing test requirements through integration with test management tools (such as Jira or TestRail). Test corpora can be uploaded in batches in formats such as CSV and JSON, with each corpus containing input prompts and expected outputs. The system preprocesses the test corpora, including cleaning, formatting, and deduplication, and stores them in a database to form a test corpus for subsequent processing and analysis.
[0093] Step S302: Based on the input prompts, expected output information, test conditions, test priority information, and preset test templates, generate multiple structured test cases, including the target test cases.
[0094] Furthermore, based on input prompts, expected outputs, test conditions, and test priority information, combined with a pre-defined test template, the system automatically generates structured test cases, including the target test cases. Structured test cases can include the following structured information: Test ID: a unique identifier for the test case; Input: user input prompts; Expected Output: the desired model output; Test Conditions: specific test conditions; Priority: the priority of the test case. The system stores the generated test cases in a test case library for subsequent test execution.
[0095] Through the above steps, this application embodiment allows test requirements to be input via a natural language interface, eliminating the need to write complex test scripts, lowering the testing threshold, enabling testers to focus on test strategy design rather than script writing, reducing the need for frequent modifications to structured test cases, adapting to model iteration updates, significantly reducing maintenance costs, and further improving testing efficiency.
[0096] In some embodiments, step S204 includes:
[0097] Step S2041: If the target expected output is fixed text, compare the model response result with the target expected output to determine the target test result.
[0098] If the expected output is fixed text, use an exact match rule. For example, if the expected output is fixed text (such as "Correct answer: 42"), then perform a direct string comparison. If the match is successful, mark the target test result as PASS; otherwise, FAIL.
[0099] Step S2042: When the expected output of the target is a fuzzy rule, calculate the target similarity between the model response result and the expected output of the target, and determine the target test result based on the target similarity.
[0100] If the expected output of the target is a fuzzy rule, use a partial matching rule. For example, if the expected output is a fuzzy rule (such as "containing the keyword name"), use NLP similarity calculation (such as BERT embedding + cosine similarity). When the similarity is greater than or equal to the threshold (such as 0.8), mark the target test result as PARTIAL_PASS, otherwise FAIL.
[0101] Step S2043: If the target expected output is executable code, dynamically run the executable code to obtain the exception judgment result, and determine the target test result based on the exception judgment result.
[0102] If the expected output is executable code, use custom validator rules (such as code generation tests). For example, if the expected output is executable code (such as Python), run the code dynamically to obtain the exception judgment result, and check whether an exception is thrown based on the exception judgment result. If no exception is thrown, mark the target test result as PASS, otherwise mark it as FAIL and record the error information.
[0103] Furthermore, the test results can also be recorded in this embodiment, specifically using structured storage, with the database table design as follows:
[0104]
[0105] Write to the test results table, example SQL:
[0106]
[0107] Furthermore, this application can also monitor and alert on test results in real time. For example, if the failure rate exceeds a threshold (e.g., 20%), it will trigger email / Slack alerts and update the results through a visualization panel (e.g., Grafana will display the pass rate trend). It can also generate preliminary reports based on the test results, such as:
[0108]
[0109] When the system malfunctions, exception handling is performed, including error classification and recovery. For network errors: automatic retry or switching to a backup API endpoint; for model errors: capturing the error code returned by the model (such as 500) and classifying it as "internal model error"; for data errors: recording dirty data samples (such as input containing NULL values) and generating a data quality report.
[0110] Through the above steps, this application embodiment implements corresponding matching rules according to different target expected outputs, making the test results more accurate and providing an intuitive test result visualization interface to help testers quickly understand the test situation and improve test decision efficiency.
[0111] In some embodiments, step S204 is followed by:
[0112] Step S205: Analyze the target test results to determine test result report information, including test pass rate information, test failure rate information, and test failure reason information.
[0113] In this embodiment, the system automatically analyzes the test results and generates test report information. The test report information includes: test pass rate information – the percentage of test cases that pass; test failure rate information – the percentage of test cases that fail; test failure reason information – analysis of the failed test cases; and detailed test results – detailed results for each test case.
[0114] Step S206: Based on the test result report information, adjust the corresponding model training data and model training parameters during the training process of the target large language model.
[0115] Furthermore, based on the test result report information, the system identifies failed test cases and analyzes the reasons for failure, such as semantic mismatch (the model output is semantically inconsistent with the expected output); format mismatch (the model output format is inconsistent with the expected format); and logical errors (the model output has logical errors). A visual interface is provided to display the test results, including charts and tables, to facilitate testers' quick understanding of the test situation. This embodiment of the application automatically optimizes test cases based on the test result report information, including: generating new test cases for failed cases; optimizing test conditions to improve test coverage; and adjusting the priority of test cases based on the test results. Improvement suggestions are also generated to help developers optimize the model, including: pointing out potential problems in the model; providing optimization suggestions; and classifying problems into semantic problems, format problems, and logical problems.
[0116] In this embodiment of the application, the test results are fed back into the model training process to achieve continuous testing and continuous optimization, including: using the test results as part of the training data, using the test results to adjust model parameters, generating test reports for the development team to refer to, etc.
[0117] Through the above steps, the embodiments of this application automatically optimize test cases based on test results, improve test coverage, and feed the test results back into the model training process to achieve continuous testing and continuous optimization, thereby further improving the accuracy and efficiency of test results.
[0118] The embodiments of this application will be described and illustrated below through preferred embodiments.
[0119] Figure 4 This is a flowchart of a model testing method according to a preferred embodiment of this application. For example... Figure 4 As shown, this model method includes the following steps:
[0120] Step S401: Obtain the model API information, database information, and UI information of the target test environment;
[0121] Step S402: Based on the model API information, database information, and UI information, access the target test environment through the model context protocol;
[0122] Step S403: Obtain the model API status information, database snapshot information, and UI structure synchronization information corresponding to the target test environment through the model context protocol;
[0123] Step S404: Based on the model API status information, database snapshot information, and UI structure synchronization information, construct the target test request according to the protocol corresponding to the model API.
[0124] Step S405: Send the target test request to the target large language model so that the target large language model outputs the model response result according to the target test request;
[0125] Step S406: Determine the target expected output in the target test case, and determine the target test result based on the model response result and the target expected output.
[0126] It should be noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.
[0127] This embodiment also provides a model testing device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the terms "module," "unit," "subunit," etc., can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.
[0128] Figure 5 This is a structural block diagram of a model testing apparatus according to an embodiment of this application, such as... Figure 5 As shown, the device includes a test environment access module 10, a test request construction module 20, a response result output module 30, and a test result determination module 40.
[0129] Test environment access module 10 is used to access the target test environment, which is configured to be compatible with the model context protocol.
[0130] The test request construction module 20 is used to obtain the target test case and the environment context information corresponding to the target test environment through the model context protocol, and construct the target test request based on the target test case and the environment context information;
[0131] The response output module 30 is used to send the target test request to the target large language model so that the target large language model outputs the model response result according to the target test request.
[0132] The test result determination module 40 is used to determine the target expected output in the target test case and to determine the target test result based on the model response result and the target expected output.
[0133] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.
[0134] This embodiment also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.
[0135] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.
[0136] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:
[0137] Connect to the target test environment, which is configured to be compatible with the model context protocol;
[0138] The target test case and the environment context information corresponding to the target test environment are obtained through the model context protocol, and the target test request is constructed based on the target test case and the environment context information;
[0139] Send the target test request to the target large language model so that the target large language model outputs the model response result according to the target test request;
[0140] Determine the target expected output in the target test cases, and determine the target test result based on the model response results and the target expected output.
[0141] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.
[0142] Furthermore, in conjunction with the model testing methods in the above embodiments, this application embodiment can provide a storage medium for implementation. This storage medium stores a computer program; when executed by a processor, the computer program implements any of the model testing methods in the above embodiments.
[0143] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0144] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.
[0145] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.
Claims
1. A model testing method characterized by comprising: The method comprises the following steps: accessing a target test environment configured to be suitable for a model context protocol; obtaining a target test case and environment context information corresponding to the target test environment through the model context protocol, and constructing a target test request according to the target test case and the environment context information; sending the target test request to a target large language model, so that the target large language model outputs a model response result according to the target test request; determining a target expected output in the target test case, and determining a target test result according to the model response result and the target expected output.
2. The model testing method according to claim 1, characterized by, The step of accessing the target test environment configured to be suitable for the model context protocol comprises: obtaining model API information, database information and UI information of the target test environment; accessing the target test environment through the model context protocol according to the model API information, the database information and the UI information, wherein the target test environment is configured to be suitable for the model context protocol.
3. The model testing method according to claim 2, characterized by, The step of obtaining the target test case and the environment context information corresponding to the target test environment through the model context protocol, and constructing the target test request according to the target test case and the environment context information comprises: obtaining model API state information, database snapshot information and UI structure synchronization information corresponding to the target test environment through the model context protocol; constructing the target test request according to the model API state information, the database snapshot information and the UI structure synchronization information according to a protocol corresponding to the model API.
4. The model testing method according to claim 3, characterized by, The step of obtaining the model API state information, the database snapshot information and the UI structure synchronization information corresponding to the target test environment through the model context protocol comprises: sending a health check request to the model API through the model context protocol, to obtain model API state information output by the model API in response to the health check request; querying the database information through the model context protocol, to determine database snapshot information including test case data; obtaining UI structure synchronization information including a current page DOM structure and a UI snapshot through the model context protocol.
5. The model testing method according to any one of claims 1 to 4, characterized in that, The method further comprises: obtaining test corpus, and parsing the test corpus to determine input prompt information, expected output information, test condition information and test priority information included in the test corpus; generating a plurality of structured test cases including a target test case according to the input prompt information, the expected output information, the test condition information, the test priority information and a preset test template.
6. The model testing method according to claim 5, wherein The step of determining the target expected output in the target test case, and determining the target test result according to the model response result and the target expected output comprises: in a case where the target expected output is fixed text, performing string comparison on the model response result and the target expected output to determine the target test result; In the case where the target expected output is a fuzzy rule, a target similarity between the model response result and the target expected output is calculated, and a target test result is determined according to the target similarity; In the case where the target expected output is executable code, the executable code is dynamically run to obtain an exception judgment result, and a target test result is determined according to the exception judgment result.
7. The model testing method according to claim 6, wherein After determining the target expected output in the target test case, and determining a target test result according to the model response result and the target expected output, the method further comprises: analyzing the target test result to determine test result report information including test pass rate information, test failure rate information, and test failure cause information; adjusting corresponding model training data and model training parameters in the target large language model training process according to the test result report information.
8. A model test device characterized by comprising: The test environment access module, the test request construction module, the response result output module, and the test result determination module are included: The test environment access module is configured to access a target test environment, and the target test environment is configured to be suitable for a model context protocol. The test request construction module is configured to obtain a target test case and environment context information corresponding to the target test environment through the model context protocol, and to construct a target test request according to the target test case and the environment context information. The response result output module is configured to send the target test request to a target large language model, so that the target large language model outputs a model response result according to the target test request. The test result determination module is configured to determine a target expected output in the target test case, and to determine a target test result according to the model response result and the target expected output. 9.An electronic device comprising a memory and a processor, the electronic device characterized by, The memory stores a computer program, and the processor is configured to run the computer program to execute the model test method of any one of claims 1 to 7.
10. A storage medium, characterized by The storage medium stores a computer program, and the computer program is configured to execute the model test method of any one of claims 1 to 7 when running.