Model testing method and device

Abstract

The invention relates to a model testing method and device. The model test method comprises the steps that a test case and a standard answer library are constructed, and the test case is compiled by adopting a ShellSpec test framework; calling the model to execute reasoning for the test case to obtain a reasoning result; obtaining performance data of the model executing reasoning to generate a performance test result; evaluating the matching degree between the reasoning result and a standard answer library to generate a precision test result; and generating a structured test report based on the performance test result and the precision test result.

Description

Technical Field

[0001] This application generally relates to the field of artificial intelligence, and more specifically, to a model testing method and apparatus. Background Technology

[0002] With the widespread application of large AI (Artificial Intelligence) models (e.g., Large Language Models, LLMs) in fields such as intelligent question answering, content generation, code programming, and logical reasoning, the accuracy, reliability, and response performance of their outputs have become key indicators for measuring their quality. Systematic, automated, and repeatable testing and evaluation are crucial during the research, development, iteration, and deployment of large AI models. Summary of the Invention

[0003] This application provides a model testing method and apparatus, which utilizes the ShellSpec Behavior-Driven Development (BDD) testing framework and combines it with an intelligent fuzzy matching algorithm to achieve automated accuracy verification, performance benchmark testing, and automatic comparison of massive question-answer pairs for the inference results of large AI models.

[0004] According to one aspect of this application, a model testing method is provided, comprising: constructing test cases and a standard answer library, wherein the test cases are written using the ShellSpec testing framework; invoking a model to perform inference against the test cases to obtain inference results; obtaining performance data of the model performing the inference to generate performance test results; evaluating the matching degree between the inference results and the standard answer library to generate accuracy test results; and generating a structured test report based on the performance test results and the accuracy test results.

[0005] According to another aspect of this application, a model testing apparatus is provided, comprising: one or more processors; and a memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the model testing method described above.

[0006] According to another aspect of this application, a non-transitory computer-readable storage medium is provided, storing instructions that, when executed by one or more processors, cause the one or more processors to perform the above-described model testing method.

[0007] According to another aspect of this application, a computer program product is provided, including instructions that, when executed by one or more processors, cause the one or more processors to perform the above-described model testing method. Attached Figure Description

[0008] Figure 1 A schematic flowchart of a model testing method according to an embodiment of this application is shown.

[0009] Figure 2 Another schematic flowchart of a model testing method according to an embodiment of this application is shown.

[0010] Figure 3 A schematic diagram of a model testing apparatus according to an embodiment of this application is shown. Detailed Implementation

[0011] The features and exemplary embodiments of various aspects of this application will now be described in detail. Numerous specific details are provided in the following detailed description to provide a thorough understanding of this application. However, it will be apparent to those skilled in the art that this application can be implemented without some of these specific details. The following description of embodiments is merely intended to provide a better understanding of this application by illustrating examples. This application is by no means limited to any specific configurations and algorithms described below, but rather covers any modifications, substitutions, and improvements to elements, components, and algorithms without departing from the spirit of this application. Well-known structures and techniques are not shown in the accompanying drawings and the following description in order to avoid unnecessarily obscuring this application.

[0012] In the research, development, iteration, and deployment of large-scale AI models, systematic, automated, and repeatable testing and evaluation are crucial. Currently, the testing and evaluation of large-scale AI models suffers from the following main shortcomings.

[0013] 1. Low testing efficiency and heavy reliance on manual testing: Existing testing methods typically require testers to manually or semi-automatically input pre-set questions into the model and manually check whether the output answers meet expectations. When faced with a massive number of test cases (e.g., tens of thousands of questions), this process is time-consuming and labor-intensive, and is prone to misjudgments and omissions due to human fatigue, failing to meet the needs of agile development and continuous delivery.

[0014] 2. Subjective and inconsistent evaluation criteria: For open-ended, creative, or answers with multiple reasonable expressions, the model output is often not unique. Different evaluators have subjective differences in their judgment of whether the answer is correct or acceptable, resulting in a lack of objectivity, consistency, and repeatability in the evaluation results, making it difficult to form a unified and quantifiable quality standard.

[0015] 3. Disjointed Accuracy and Performance Testing: Traditional testing processes typically separate testing of a model's functional correctness (accuracy) from testing its inference speed and resource consumption (performance). This results in lengthy testing procedures, fragmented toolchains, and makes it difficult to perform correlation analysis and regression monitoring of the model's overall "accuracy-performance" performance under a unified benchmark. For example, version upgrades may improve accuracy while causing performance degradation, and fragmented testing makes it difficult to quickly and intuitively reveal this trade-off.

[0016] 4. Poor readability and high maintenance costs of test cases: Test cases (e.g., question-expected answer pairs) are often in the form of script code, configuration files, or unstructured documents, lacking a clear and unified expression paradigm. This not only makes the test intent obscure and difficult to understand, hindering team collaboration, knowledge transfer, and new member comprehension, but also makes the reuse, expansion, and maintenance of test cases extremely difficult.

[0017] This application provides a model testing method and apparatus that can be used for various large AI models. It can integrate accuracy verification, performance benchmark testing, and intelligent answer comparison into a single / automated process, thereby improving the efficiency, objectivity, and maintainability of large AI model testing.

[0018] The present application will now be described in further detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the application to these specific forms.

[0019] Figure 1 A schematic flowchart of a model testing method 100 according to an embodiment of this application is shown. Figure 1 As shown, the model testing method 100 may include the following steps S101-S105.

[0020] In step S101, the model testing method 100 may include constructing test cases and a standard answer library. In some embodiments, step S101 may include writing test cases using the ShellSpec testing framework.

[0021] In some embodiments, test cases may be formatted using a Given-When-Then structure. For example, based on Behavior-Driven Development (BDD) syntax, the problem to be tested, expected results, and performance benchmark thresholds can be written and managed in a Given (given conditions), When (action to be performed), Then (expected results) structure to build test cases and a standard answer library. In this context, Given can be used to indicate the given conditions for model testing, such as loading the model, preparing test data, etc.; When can be used to indicate the execution of the test action, i.e., calling the model for inference; and Then can be used to indicate the verification results, which may include at least two aspects: accuracy and performance.

[0022] In some embodiments, the standard answers can be directly embedded in the test cases, or they can be stored in an external file (e.g., JSON, YAML) or database and loaded during the Given phase. This allows for the construction of a maintainable standard answer library.

[0023] In step S102, the model testing method 100 may include invoking the model to perform inference for the test cases to obtain the inference result of the model performing the inference.

[0024] For example, you can call the application programming interface (API) of the AI ​​model to be tested, input test cases into the AI ​​model to perform inference, and obtain the inference result of the AI ​​model performing the inference.

[0025] In step S103, the model testing method 100 may include acquiring performance data of the model performing inference to generate performance test results.

[0026] In some embodiments, performance data for model inference may include, but is not limited to, at least one of the following: memory usage, inference response time, inference throughput, inference latency, token usage, and token generation rate. For example, inference response time may indicate the total time elapsed from sending a test request to receiving the complete inference output; token usage may indicate the number of tokens used by the model to perform inference; and inference throughput / latency may indicate the number of requests per second (QPS) or average latency in a batch testing scenario.

[0027] In some embodiments, step S103 may be performed after step S102; alternatively, step S103 may be performed synchronously with step S102.

[0028] In some embodiments, step S103 may include comparing the performance data of the model performing inference with historical benchmark performance data. For example, the performance data of the model performing inference may be compared with historical benchmark performance data (or a predefined performance benchmark value) to determine whether there is performance regression or whether the service level agreement (SLA) requirements are met.

[0029] For example, if a performance metric (such as the response time of a critical interface) degrades beyond a preset "tolerance (or threshold)" (e.g., response time increases by more than 10%), the test case can not only be marked as "performance failed" in the test report (described below), but an alert can also be automatically triggered to indicate that there may be a performance regression.

[0030] In some embodiments, performance test results may include structured objects. For example, performance test results may include, but are not limited to, the following: • test_case_id: Test case identifier; • Metrics: Performance data, such as {"response_time_ms": 1520, "total_tokens": 45}; • baseline_metrics: Historical baseline values, such as {"response_time_ms": 1200, "total_tokens": 40}; • regression_status: Regression status, such as "PASS", "SLOWDOWN", "REGRESSION", etc. • comparison: Details of the comparison with historical benchmarks, such as {"response_time_change": "+26.7%", "is_regression": true}.

[0031] In step S104, the model testing method 100 may include evaluating the matching degree between the model's reasoning results and a standard answer library to generate accuracy test results.

[0032] In some embodiments, evaluating the matching degree between the model's inference results and the standard answer library may include a multi-level matching strategy. In some embodiments, for example, step S104 may include performing an exact matching algorithm and / or an intelligent fuzzy matching algorithm on the model's inference results and the standard answer library.

[0033] For example, an exact matching algorithm can be executed first on the model's inference results and a standard answer database to check whether the model's inference results are the same as the standard answer database, or to check whether the model's inference results conform to preset regular expressions, keyword lists, etc. The exact matching algorithm is relatively fast and suitable for application scenarios where a definite answer exists.

[0034] If an exact match fails, an intelligent fuzzy matching algorithm can be executed based on the model's inference results and the standard answer library. In some embodiments, the intelligent fuzzy matching algorithm may include, but is not limited to, at least one of the following: cosine similarity matching algorithm, edit distance matching algorithm, Sentence-BERT matching algorithm, keyword / semantic rule-based matching algorithm, etc. The matching algorithms listed above are merely examples of intelligent fuzzy matching algorithms that can be used in the embodiments of this application, and this application is not limited thereto.

[0035] In some embodiments, in addition to exact matching algorithms and intelligent fuzzy matching algorithms, deep semantic matching can also be performed. For example, if the matching degree evaluation results of both the exact matching algorithm and the intelligent fuzzy matching algorithm are below a threshold, a deep learning-based semantic model can be executed. For example, the inference results of the model and a standard answer library can be input into a trained Sentence Transformer model to obtain the corresponding high-dimensional semantic vectors, and the cosine similarity between the high-dimensional semantic vectors can be calculated.

[0036] In some embodiments, accuracy test results may also include structured objects. For example, accuracy test results may include, but are not limited to, the following: • test_case_id: Test case identifier; •question: Enter your question; • expected_answer: The expected (standard) answer; •model_output: The actual output of the model; • match_algorithm: The matching algorithm used; •similarity_score: The calculated matching score (can be between 0 and 1); • pass_status: Test result, based on similarity_score >= threshold_used to determine if a match is found; •threshold_used: The matching threshold used by the test case.

[0037] The following explanation uses the cosine similarity matching algorithm as an example to illustrate the process of obtaining accuracy test results. For instance, the cosine similarity matching algorithm may include the following steps: 1. Text preprocessing: Preprocessing the model's actual output (model_output) and expected answer (expected_answer), for example, cleaning; 2. Vectorization: Convert the preprocessed text into numerical vectors, where TF-IDF or a simpler bag-of-words model can be used; 3. Calculate cosine similarity: Obtain the similarity score; 4. Result determination: Compare the similarity score with the preset threshold (threshold_used, such as 0.85) to obtain the determination result (e.g., match or no match).

[0038] In some embodiments, the model testing method 100 may further include triggering a manual review process if the matching degree between the model's inference result and the standard answer library is lower than a preset threshold. For example, when the similarity score of the test case is lower than a preset threshold (or the minimum acceptable threshold), the matching result can be determined as "fuzzy" or "suspicious," and in this case, a manual review process can be triggered.

[0039] For example, to trigger a manual review process, a review task can be created in an internal work order system or collaboration platform. This task may include, but is not limited to, test case details, model output, expected answer, similarity score, and explanation of the matching algorithm. Based on preset rules, the created review task is assigned to relevant reviewers (e.g., domain experts, test leads) and a notification is sent. For instance, a human-computer interaction interface can be provided to display the matching results to reviewers, allowing them to choose options such as "pass," "fail," or "modify expected answer." The review results can be used to update test results. For example, if "modify expected answer" is selected, the standard answer library is automatically updated, and the relevant test cases may be re-executed.

[0040] This human review process is particularly suitable for scenarios that are highly creative, highly subjective, or involve complex logical reasoning, in order to ensure the final accuracy of the assessment and to continuously optimize the matching algorithm and thresholds using human feedback.

[0041] In step S105, the model testing method 100 may include generating a structured test report based on performance test results and accuracy test results.

[0042] In some embodiments, a structured test report may include, but is not limited to, at least one of the following: accuracy test pass rate, performance test pass rate, and execution status of the test cases.

[0043] Considering the structured objects included in the performance and accuracy test results as described above, ShellSpec can generate a structured test report after the tests are executed. This report aggregates performance test results (e.g., metrics, regression_status, etc.) and accuracy test results (e.g., similarity_score, pass_status, etc.) for the test cases. For example, structured test report files (e.g., JSON, HTML, Markdown) can be generated using a custom test report generator (e.g., a Python script).

[0044] In some embodiments, a structured test report can be a multi-level document. For example, a structured test report may include, but is not limited to: a summary, which may include the total number of test cases, pass rate (accuracy test, performance test), and execution status (pass / fail); accuracy details, which may include a list of failed test cases (e.g., test question, model output, expected answer, and matching score), a histogram of matching score distribution, etc.; performance details, which may include average response time, total token consumption, and performance comparison with the previous test or historical benchmark performance data, etc.; and comprehensive analysis and suggestions, which may indicate test cases where accuracy and performance may be correlated (e.g., answering a complex question incorrectly and taking a long time), and provide improvement suggestions, such as "optimize the prompt words of module X" or "focus on performance bottlenecks in scenario Y," etc.

[0045] In addition, a structured test report may include a raw data appendix containing all the details of all test cases for further analysis and recommendations.

[0046] In some embodiments, the model testing method 100 may optionally include analyzing and visualizing the structured test report. For example, after generating a structured test report in formats such as JSON, HTML, or Markdown, the structured test report can be further analyzed and visualized through a front-end interface.

[0047] For example, visual dashboards can be created to dynamically display various test metrics, such as real-time accuracy / performance test pass rates, performance trend charts (regarding multiple tests), response time distribution charts, etc. Interactive information retrieval can be implemented, such as clicking on charts or numbers in the summary of a structured test report to view specific failed test cases and their detailed information. Historical information comparison can be provided, such as offering a timeline slider to facilitate comparison of test results differences between different batches and different model versions. The above analysis and visualization methods are merely examples, and this application is not limited to them.

[0048] In some embodiments, the model testing method 100 may optionally include storing the model's inference performance data as historical benchmark performance data after generating a structured test report. In this way, the benchmark performance data can be dynamically updated to reflect changes in inference test performance in real time.

[0049] In the model testing method 100 according to the embodiments of this application, the ShellSpec Behavior-Driven Development (BDD) testing framework is used, and combined with the intelligent fuzzy matching algorithm, to realize automated accuracy verification, performance benchmark testing, and automatic comparison of massive question-answer pairs for the inference results of large AI models.

[0050] The model testing method 100 according to embodiments of this application includes one or more of the following features: 1. Automated accuracy verification: By predefining "question-expected answer" pairs, the model inference is automatically executed, and the actual output of the model is compared with the expected answer, without the need for manual checking; 2. Integrated performance benchmark testing: In the same testing process, key performance indicators such as the time consumption and word usage of each question inference are obtained synchronously and automatically compared with historical benchmark values ​​to monitor performance regression; 3. Support for massive data and intelligent fuzzy matching: An intelligent fuzzy matching algorithm based on text similarity (e.g., cosine similarity, edit distance, Sentence-BERT embedding) or keyword / semantic rules is introduced to cope with the diversity of model outputs and achieve automated evaluation of non-strict literal matching; 4. Improved readability and stability of test cases: Using ShellSpec's BDD syntax, test cases are written in a clear and structured natural language style (Given-When-Then) to make the test cases more readable and stable. Figure 1 With clear visuals, the test cases themselves become living documents, greatly improving maintainability and team collaboration efficiency.

[0051] Figure 2 Another schematic flowchart of a model testing method 200 according to an embodiment of this application is shown. It will be understood that the model testing method 200 may correspond to the above-described combination. Figure 1 The model testing method described is 100. For example... Figure 2 As shown, the model testing method 200 may include steps S201-S206.

[0052] In step S201, the model testing method 200 may include constructing ShellSpec test cases (e.g., Given-When-Then structure) and a core standard answer library. Details regarding the ShellSpec test cases and the standard answer library can be found in the description of step S101 of the model testing method 100 above. For example, step S201 may correspond to the test preparation phase.

[0053] In step S202, the model testing method 200 may include the ShellSpec test execution engine calling the API of the AI ​​model (e.g., DeepSeek, ChatGPT, etc.) to perform inference against the test cases.

[0054] In step S203, the model testing method 200 may include collecting performance data of the model's inference execution to generate performance test results. Performance data may include, but is not limited to: memory usage, inference response time, inference throughput, inference latency, token usage, and token generation rate. Details regarding the performance test results can be found in the above description of step S103 of the model testing method 100.

[0055] In step S204, model testing method 200 may include evaluating the matching degree (e.g., intelligent fuzzy matching) to generate accuracy test results. Details regarding the accuracy test results can be found in the above description of step S104 of model testing method 100. Steps S202-S204 may correspond to the automated testing and evaluation phase.

[0056] In step S205, the model testing method 200 may include generating a structured test report (JSON, HTML, Markdown, etc.) based on the performance test results generated in step S203 and the accuracy test results generated in step S204. Details regarding the structured test report can be found in the above description of step S105 of the model testing method 100.

[0057] In step S206, model testing method 200 may include test result analysis and visualization. Steps S205-S206 may be used for test result output and decision-making stages.

[0058] For specific details regarding model testing method 200, please refer to the above description of model testing method 100, which will not be repeated here. It will be understood that model testing method 200 can achieve the same beneficial effects as model testing method 100.

[0059] Figure 3 A schematic diagram of a model testing apparatus according to an embodiment of this application is shown. This model testing apparatus is shown as a computing device 300, which can be used to perform the model testing methods 100 / 200 described above. Figure 3 As shown, computing device 300 may include bus 302 or other communication mechanism for transmitting information, and one or more hardware processors 304 coupled to bus 302 for processing information. The one or more hardware processors 304 may include, for example, one or more general-purpose microprocessors.

[0060] like Figure 3As shown, in some embodiments, computing device 300 may further include main memory 306 coupled to bus 302. Main memory 306 is used to store information and instructions executed by one or more processors 304, such as random access memory (RAM), cache, and / or other dynamic storage devices. Main memory 306 may also be used to store temporary variables or other intermediate information during the execution of instructions executed by one or more processors 304. When these instructions are stored in storage media accessible to one or more processors 304, they can cause computing device 300 to become a dedicated machine customized to perform the operations specified in the instructions. Storage device 308 may include non-volatile and / or volatile storage media. Non-volatile storage media may include, for example, optical discs or magnetic disks. Volatile storage media may include dynamic memory. Common forms of storage media may include, for example, floppy disks, hard disks, solid-state drives, magnetic tape, or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a perforated pattern, RAM, DRAM, PROM, and EPROM, FLASH-EPROM, NVRAM, any other memory chip or cartridge, or their networking versions.

[0061] like Figure 3 As shown, in some embodiments, computing device 300 may further include one or more communication interfaces or network interfaces 310 coupled to bus 302. Network interface 310 may provide bidirectional data communication coupling to one or more network links connected to one or more networks. As another example, network interface 310 may be a local area network (LAN) card to provide data communication connectivity to a LAN-compatible (or WAN component communicating with a WAN) network. Wireless links may also be implemented.

[0062] The execution of certain operations can be distributed across processors rather than residing within a single machine, but rather deployed across multiple machines. In some example embodiments, the processor or processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other example embodiments, the processor or processor-implemented engine may be distributed across multiple geographic locations.

[0063] Each of the processes, methods, and algorithms described above may be embodied in code modules executed by one or more computer systems or computer processors including computer hardware, and may be fully or partially automated by these code modules. The processes and algorithms may be implemented partially or fully in dedicated circuit systems.

[0064] When the functions disclosed herein are implemented as software functional units and sold or used as stand-alone products, they may be stored in a processor-executable, non-volatile, computer-readable storage medium. Specific technical solutions (all or part) disclosed herein, or aspects contributing to the prior art, may be embodied in the form of a software product. The software product may be stored in a storage medium and includes several instructions that cause a computing device (which may be a personal computer, server, network device, etc.) to perform all or some steps of the methods of the embodiments of this application. The storage medium may include a flash drive, portable hard disk drive, ROM, RAM, magnetic disk, optical disk, other media operable to store program code, or any combination thereof.

[0065] Specific embodiments further provide an apparatus including a processor and a non-transitory computer-readable storage medium storing instructions executable by the processor to cause the apparatus to perform operations corresponding to steps in any method of the embodiments disclosed above. Specific embodiments further provide a non-transitory computer-readable storage medium storing instructions executable by one or more processors to cause the one or more processors to perform operations corresponding to steps in any method of the embodiments disclosed above.

[0066] The embodiments disclosed herein can be implemented via a cloud platform, server, or server cluster (collectively referred to below as the "Service System") that interacts with a client. The client can be a terminal device or a client registered by a user at the platform, wherein the terminal device can be a mobile terminal, a personal computer (PC), or any device that can have the platform application installed.

[0067] The various features and processes described above can be used independently of each other or combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this disclosure. Additionally, certain method or process blocks may be omitted in some embodiments. The methods and processes described herein are not limited to any particular order, and their associated blocks or states may be executed in other suitable orders. For example, described blocks or states may be executed in an order other than that specifically disclosed, or multiple blocks or states may be combined into a single block or state. Example blocks or states may be executed sequentially, in parallel, or in some other manner. Blocks or states may be added to or removed from the disclosed example embodiments. The exemplary systems and components described herein may be configured differently than described. For example, components may be added to, removed from, or rearranged compared to the disclosed example embodiments.

[0068] The various operations of the exemplary methods described herein can be performed at least in part by an algorithm. The algorithm may be included in program code or instructions stored in memory (e.g., the aforementioned non-transitory computer-readable storage medium). This algorithm may include a machine learning algorithm. In some embodiments, the machine learning algorithm may not explicitly refer to the computer as performing the function but may learn from training data to generate a predictive model of the function.

[0069] The various operations of the exemplary methods described herein can be performed, at least in part, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, these processors can constitute an engine of processor implementation that operates to perform one or more of the operations or functions described herein.

[0070] Similarly, the methods described herein may be implemented at least in part by a processor, wherein one or more specific processors are instances of hardware. For example, at least some operations of the methods may be performed by one or more processors or an engine implemented by a processor. Furthermore, one or more processors may also be operable to support the execution of relevant operations in a “cloud computing” environment or as the execution of relevant operations in a “Software as a Service” (SaaS) context. For example, at least some operations may be performed by a group of computers (as an example of a machine containing processors), wherein these operations are accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., application programming interfaces (APIs)).

[0071] The execution of certain operations can be distributed across processors rather than residing within a single machine, and can be deployed across multiple machines. In some example embodiments, the processor or processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other example embodiments, the processor or processor-implemented engine may be distributed across multiple geographic locations.

[0072] Throughout this specification, multiple instances may be implemented as components, operations, or structures described as a single instance. Although individual operations of one or more methods are illustrated and described as separate operations, one or more of these individual operations may be performed simultaneously, and not necessarily in the order illustrated. Structures and functions presented as separate components in the example configuration may be implemented as composite structures or components. Similarly, structures and functions presented as single components may be implemented as single components. These and other variations, modifications, additions, and improvements fall within the scope of this document.

[0073] As used herein, "or" is inclusive rather than exclusive unless explicitly indicated by the context. Therefore, in this document, "A, B, or C" means "A, B, A and B, A and C, B and C, or A, B, and C" unless explicitly indicated by the context. Furthermore, "and" is combined and separate unless explicitly indicated by the context. Therefore, in this document, "A and B" means "A and B, combined or separate" unless explicitly indicated by the context. Additionally, multiple instances of resources, operations, or structures described herein may be provided as a single instance. Furthermore, the boundaries between various resources, operations, engines, and data storage devices are somewhat arbitrary and specific operations are illustrated within the context of a particular illustrative configuration. Other functional assignments are foreseeable and fall within the scope of various embodiments of this disclosure. Generally, structures and functions presented as individual resources in example configurations may be implemented as combined structures or resources. Similarly, structures and functions presented as single resources may be implemented as single resources. These and other changes, modifications, additions, and improvements fall within the scope of the embodiments of this disclosure as expressed in the appended claims. Therefore, this specification and drawings should be considered illustrative rather than restrictive.

[0074] The terms “comprising” or “including” are used to indicate the presence of a subsequently claimed feature, but do not preclude the addition of other features. Unless otherwise specifically stated or otherwise understood in the context in which they are used, conditional language such as “may,” “can,” “may,” and “can” is generally intended to convey that certain embodiments include certain features, components, and / or steps that are not included in other embodiments. Therefore, this conditional language is generally not intended to imply that one or more embodiments require features, components, and / or steps in any way, or that one or more embodiments must include logic for determining whether such features, components, and / or steps are included in or performed in any particular embodiment, with or without user input or prompts.

[0075] Although the general outline of the subject matter has been described with reference to specific exemplary embodiments, various modifications and changes may be made to these embodiments without departing from the broad scope of embodiments of this disclosure. Where more than one embodiment is disclosed, these embodiments of the subject matter may be referred to individually or collectively herein as the term "invention," this is for convenience only and is not intended to automatically limit the scope of this application to any single disclosure or concept.

[0076] The embodiments illustrated herein are described in detail to enable those skilled in the art to practice the disclosed teachings. Other embodiments may be used and derived therefrom, such that structural and logical substitutions and changes may be made without departing from the scope of this disclosure. Therefore, “implementation” is not intended to be limiting, and the scope of the various embodiments is defined only by the appended claims and their full scope.

Claims

1. A model testing method, characterized in that, include: Build a test case and standard answer library, wherein the test cases are written using the ShellSpec testing framework; The model is invoked to perform inference for the test cases to obtain the inference results; Obtain performance data of the model performing the inference to generate performance test results; Evaluate the matching degree between the reasoning results and the standard answer database to generate accuracy test results; and Based on the performance test results and the accuracy test results, a structured test report is generated.

2. The method according to claim 1, characterized in that, Also includes: The structured test report is analyzed and visualized.

3. The method according to claim 1, characterized in that, The performance data includes at least one of the following: memory usage, inference response time, inference throughput, inference latency, token usage, and token generation rate.

4. The method according to claim 1, characterized in that, The test cases are formatted using a Given-When-Then structure.

5. The method according to claim 1, characterized in that, Evaluating the matching degree between the reasoning result and the standard answer database includes: The exact matching algorithm and / or intelligent fuzzy matching algorithm are executed on the reasoning result and the standard answer library.

6. The method according to claim 5, characterized in that, The intelligent fuzzy matching algorithm includes at least one of the following: cosine similarity matching algorithm, edit distance matching algorithm, Sentence-BERT matching algorithm, and keyword / semantic rule-based matching algorithm.

7. The method according to claim 1, characterized in that, The generation of the performance test results includes: The performance data of the model performing the inference is compared with historical benchmark performance data.

8. The method according to claim 1, characterized in that, Also includes: After generating the structured test report, the performance data of the model performing the inference is stored as historical benchmark performance data.

9. The method according to claim 1, characterized in that, Also includes: If the matching degree between the reasoning result and the standard answer database is lower than a preset threshold, a manual review process is triggered.

10. The method according to claim 1, characterized in that, The structured test report includes at least one of the following: accuracy test pass rate, performance test pass rate, and execution status of the test cases.

11. A model testing device, characterized in that, include: One or more processors; as well as A memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the method according to any one of claims 1-10.

12. A non-transitory computer-readable storage medium storing instructions, characterized in that, When executed by one or more processors, the instructions cause the one or more processors to perform the method according to any one of claims 1-10.

13. A computer program product, comprising instructions, characterized in that, When executed by one or more processors, the instructions cause the one or more processors to perform the method according to any one of claims 1-10.

Images