Model testing method, device, equipment and storage medium

Abstract

The application discloses a model testing method and device, equipment and a storage medium, and relates to the technical field of artificial intelligence. The method comprises the following steps: extracting a test prompt word of a to-be-tested model from a test instruction of a user; constructing a test corpus set corresponding to the test prompt word through a preset knowledge graph, inputting the test corpus set into the to-be-tested model, and obtaining a model output result; obtaining an expected result of the test corpus set from a baseline corpus library, wherein the baseline corpus library comprises a mapping relationship between each corpus and a preset standard answer; calling a large language model to analyze the expected result and the model output result, and obtaining a test result of the to-be-tested model. Compared with an existing model testing method, the application can automatically generate a test corpus set that can be associated with a baseline corpus through a test prompt word driving a preset knowledge graph, thereby not only reducing the workload of manually writing a corpus, but also improving the reuse rate of corpora among different model projects, so that the overall efficiency of the model testing process is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a model testing method, apparatus, device, and storage medium. Background Technology

[0002] Currently, in artificial intelligence (AI) model software testing scenarios (such as functional acceptance testing of intelligent customer service robots, interface regression testing, etc.), in order to verify the accuracy and stability of the model's output, it is generally necessary to construct a corpus and call the model interface for comparison.

[0003] Current testing methods typically involve testers writing corpus files and then calling the model under test one by one through an API debugging tool, after which the model returns the actual results. For example, testers might need to manually write dozens of different questions for an order refund scenario, and each corpus requires manually filling in request parameters and sending request data.

[0004] However, the entire process relies on offline document management and manual operation. The corpus is generally only applicable to the current project. If testing different model projects, the corpus needs to be rewritten. For example, when it is necessary to test the model's ability to respond to user emotions, testers need to rewrite a completely new corpus, and the corpus from the previous order refund scenario is difficult to use, resulting in low efficiency in the model testing process. Summary of the Invention

[0005] The main objective of this application is to provide a model testing method, apparatus, device, and storage medium, which aims to solve the technical problem of low efficiency in existing model testing processes.

[0006] To achieve the above objectives, this application proposes a model testing method, which includes: Extract test prompts corresponding to the model to be tested from the user's test instructions; A test corpus corresponding to the test prompt words is constructed by using a pre-defined knowledge graph, and the test corpus is input into the model to be tested to obtain the model output results; Obtain the expected results corresponding to the test corpus from the baseline corpus, which includes the mapping relationship between each corpus and the preset standard answer; The large language model is invoked to analyze the expected results and the output results of the model, and the test results of the model to be tested are obtained.

[0007] In one embodiment, the step of constructing a test corpus set corresponding to the test prompt words through a preset knowledge graph includes: The large language model is invoked to construct a corpus of the test prompt words, generating raw corpus data. The quality of the original corpus data is verified based on a preset knowledge graph. Each piece of text in the original text data that has passed the verification is associated with the corresponding interface of the model to be tested to obtain a baseline identifier; Based on the baseline identifier and the original corpus data after verification, the test corpus set of the model to be tested is determined.

[0008] In one embodiment, the step of inputting the test corpus into the model to be tested and obtaining the model output includes: The test corpus is classified to obtain at least two target corpus sets, and a preset test round is determined based on the target corpus sets. A request body is constructed based on the target corpus set of the current test round, and the request body is sent to the model to be tested to obtain the current output result; Determine whether the current test round has reached the preset test round; If the current test round has not reached the preset test round, return to the step of constructing the request body based on the target corpus set of the current test round, until the current test round reaches the preset test round, and obtain the model output result.

[0009] In one embodiment, the step of constructing a request body based on the target corpus set of the current test round and sending the request body to the model to be tested to obtain the current output result includes: Obtain the baseline identifier of the target corpus set in the current test round, and determine the interface configuration information and request body template corresponding to the model to be tested based on the baseline identifier; The placeholder positions for request parameters in the request body template are determined based on the interface configuration information. Obtain the content of each corpus in the target corpus, and fill the corpus content into the request body template based on the placeholder position to obtain at least two completed request bodies; Each request body is sent to the model under test in an asynchronous, multi-threaded manner to obtain the current output result.

[0010] In one embodiment, the step of calling a large language model to analyze the expected result and the model output to obtain the test result of the model to be tested includes: The expected results are compared with the model output to obtain the differential results, and the differential corpus corresponding to the differential results is determined. The large language model is invoked to perform root cause analysis on the expected results and corresponding model output results of the differentiated corpus, so as to obtain the root cause of failure for each of the differentiated corpus entries; The root causes of failure and the differentiated corpus are used as the test results for the model under test.

[0011] In one embodiment, after the step of calling the large language model to analyze the expected result and the model output to obtain the test result of the model to be tested, the method further includes: Extract the performance metrics of the model under test in multiple historical test rounds; The large language model is invoked to analyze the time-series changes of the performance indicators and obtain the performance change trend; Based on the root causes of failure and the performance change trend, a corpus supplementation strategy for the test corpus is determined, and a model iteration strategy for the model to be tested is determined based on the performance change trend. Based on the corpus supplementation strategy and the model iteration strategy, a strategy optimization report corresponding to the test results is generated.

[0012] In one embodiment, the step of determining the model iteration strategy of the model to be tested based on the performance change trend includes: Obtain the test context information of the model to be tested in the historical test rounds; Based on the performance change trend, determine the time-series change trajectory of the performance index in the historical test rounds; Based on the root cause of failure and the time-series change trajectory, the large language model is invoked to generate an optimization action set, which includes a model hyperparameter adjustment strategy, a training data supplementation strategy, and an interface call chain investigation strategy. Based on the model hyperparameter adjustment strategy, the training data supplementation strategy, and the interface call chain investigation strategy, a model iteration strategy for the model to be tested is generated.

[0013] Furthermore, to achieve the above objectives, this application also proposes a model testing apparatus, the apparatus comprising: The prompt word module is used to extract the test prompt words corresponding to the model to be tested from the user's test instructions; The corpus module is used to construct a test corpus set corresponding to the test prompt words through a preset knowledge graph, and input the test corpus set into the model to be tested to obtain the model output results; The expected module is used to obtain the expected results corresponding to the test corpus from the baseline corpus, which includes the mapping relationship between each corpus and the preset standard answer; The analysis module is used to call the large language model to analyze the expected results and the model output results to obtain the test results of the model to be tested.

[0014] In addition, to achieve the above objectives, this application also proposes a model testing device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the model testing method as described above.

[0015] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the model testing method described above.

[0016] One or more technical solutions proposed in this application have at least the following technical effects: The model testing method of this application includes: extracting test prompt words corresponding to the model to be tested from the user's test instructions; constructing a test corpus set corresponding to the test prompt words through a preset knowledge graph, and inputting the test corpus set into the model to be tested to obtain the model output result; obtaining the expected result corresponding to the test corpus set from a baseline corpus, wherein the baseline corpus includes the mapping relationship between each corpus and a preset standard answer; and calling a large language model to analyze the expected result and the model output result to obtain the test result of the model to be tested.

[0017] This application first extracts test prompt words corresponding to the model under test from the user's test instructions, constructs a test corpus set corresponding to the test prompt words through a pre-set knowledge graph, then obtains the expected results corresponding to the test corpus set from the baseline corpus, and finally calls a large language model to analyze the expected results and model output results. Compared with existing testing methods that rely on manually written corpora and whose corpora are difficult to reuse across projects, this application can automatically generate a test corpus set that can be associated with the baseline corpus by driving a pre-set knowledge graph through test prompt words. This not only reduces the workload of manually writing corpora but also improves the reusability of corpora between different model projects, thereby improving the overall efficiency of the model testing process. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the first embodiment of the model testing method of this application. Figure 2 This is a schematic diagram of the corpus generation process provided in Embodiment 1 of this application; Figure 3 This is a flowchart illustrating Embodiment 2 of the model testing method of this application; Figure 4 This is a schematic diagram of the model response provided in Embodiment 2 of this application; Figure 5 This is a flowchart illustrating the third embodiment of the model testing method of this application. Figure 6 This is a flowchart illustrating the report interpretation process provided in Embodiment 3 of this application; Figure 7 This is a block diagram of the module structure of the model testing device according to an embodiment of this application; Figure 8 This is a schematic diagram of the hardware operating environment involved in the model testing equipment in this application embodiment.

[0021] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0023] The main solution of this application is: Currently, in artificial intelligence (AI) model software testing scenarios (such as functional acceptance testing of intelligent customer service robots, interface regression testing, etc.), in order to verify the accuracy and stability of the model's output, it is generally necessary to construct a corpus and call the model interface for comparison.

[0024] Current testing methods typically involve testers writing corpus files and then calling the model under test one by one through an API debugging tool, after which the model returns the actual results. For example, testers might need to manually write dozens of different questions for an order refund scenario, and each corpus requires manually filling in request parameters and sending request data.

[0025] However, the entire process relies on offline document management and manual operation. The corpus is generally only applicable to the current project. If testing different model projects, the corpus needs to be rewritten. For example, when it is necessary to test the model's ability to respond to user emotions, testers need to rewrite a completely new corpus, and the corpus from the previous order refund scenario is difficult to use, resulting in low efficiency in the model testing process.

[0026] To address the aforementioned issues, this application provides a model testing method. This method first extracts test prompt words corresponding to the model under test from the user's test instructions, constructs a test corpus set corresponding to the test prompt words using a pre-defined knowledge graph, then obtains the expected results corresponding to the test corpus set from the baseline corpus, and finally calls a large language model to analyze the expected results and the model output. Compared to existing testing methods that rely on manually written corpora and whose corpora are difficult to reuse across projects, this application can automatically generate a test corpus set that can be associated with the baseline corpus by driving a pre-defined knowledge graph using test prompt words. This not only reduces the workload of manually writing corpora but also improves the reusability of corpora between different model projects, thereby improving the overall efficiency of the model testing process.

[0027] It should be noted that the executing entity of this application embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a test server, a personal computer, or other model testing device capable of executing the model testing method of this application. This embodiment does not limit this. The following uses a model testing device (hereinafter referred to as the device) as an example to describe this embodiment and the following embodiments.

[0028] Based on this, this application proposes a model testing method in the first embodiment, referring to... Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the model testing method of this application. In this embodiment, the model testing method may include steps S10 to S40: Step S10: Extract the test prompt words corresponding to the model to be tested from the user's test instructions.

[0029] It should be noted that the test command can be a command issued by the user through the human-computer interaction interface to start the model testing process. The test command may contain information to identify the model to be tested and its testing requirements.

[0030] It should also be noted that the model to be tested can be the target artificial intelligence model being tested, such as a natural language processing model, an image recognition model, or a generative large language model.

[0031] Understandably, test prompts can be text information parsed from test instructions, used to guide the generation of test corpora. For example, "medical questions and answers" in a user's input "Please test the model's accuracy in answering medical questions" would be a test prompt.

[0032] In actual use, after receiving a test command initiated by the user, the device first parses the test command and extracts the test prompt words corresponding to the model under test. Specifically, preset fields or semantic fragments in the test command can be identified through string matching, regular expressions, or natural language processing, and the identified content is used as the test prompt words. This embodiment does not impose any limitations on this.

[0033] Step S20: Construct a test corpus set corresponding to the test prompt words through a preset knowledge graph, and input the test corpus set into the model to be tested to obtain the model output results.

[0034] It is also understood that the preset knowledge graph can be a pre-built structured knowledge network used to describe AI testing strategies and corpus generation rules. This preset knowledge graph can contain the relationships between different testing scenarios, corpus types, field mapping relationships, and quality verification rules.

[0035] For example, for the test scenario of the natural language processing model, the preset knowledge graph can define the slot types, synonym expansion rules and boundary condition requirements that the corpus should contain in the "intent recognition" scenario. This embodiment does not impose any restrictions on this.

[0036] It should be understood that the test corpus can be a set of input data used to perform functional verification on the model to be tested. The test corpus contains at least one test corpus, and each test corpus can be a text, a request body, or a set of parameter values.

[0037] It should also be understood that the model output can be the response data returned by the model after receiving the test corpus as input and performing model calculations.

[0038] In practical use, after extracting the test prompt words, the device can call the corpus generation rules and testing strategies stored in the preset knowledge graph. Taking the test prompt words as input, it generates multiple corpus data that meet the testing requirements through rule matching or knowledge reasoning, and combines the generated corpus data into a test corpus set. Then, the constructed test corpus set is input into the model under test one by one or in batches, and then the device receives the response content returned by the model under test for each test corpus.

[0039] For example, assuming the test prompt is "book a flight," the pre-defined knowledge graph defines that the corpus for the "book a flight" scenario should include three slots: departure point, destination, and date, as well as synonym replacement rules. The device can generate three test corpora based on this knowledge graph: "Book me a flight from Beijing to Shanghai," "I want to book a flight from Shenzhen to Chengdu tomorrow," and "Book a flight from Guangzhou to Hangzhou the day after tomorrow." These three corpora are then used as the test corpus set and sequentially input into the intent recognition model, which serves as the model under test. The intent recognition model returns a recognition result for each corpus; for example, the first result returns "Intent: Book a flight, Departure point: Beijing, Destination: Shanghai, Date: Unspecified." Finally, these returned results are collected as the model's output.

[0040] Furthermore, in order to obtain the aforementioned test corpus set, in this embodiment, the step of constructing the test corpus set corresponding to the test prompt words through a preset knowledge graph includes: calling a large language model to construct corpus data for the test prompt words, generating raw corpus data; performing quality verification on the raw corpus data according to the preset knowledge graph; associating each piece of corpus data in the verified raw corpus data with the corresponding interface of the model to be tested to obtain a baseline identifier; and determining the test corpus set of the model to be tested based on the baseline identifier and the verified raw corpus data.

[0041] It should be noted that the original corpus data can be an unverified initial corpus set directly generated by a large language model based on test prompt words.

[0042] After obtaining the raw corpus data, the device can perform quality verification on the raw corpus data. This process can be executed based on the verification rules stored in the preset knowledge graph to ensure the integrity, diversity, domain adaptability and format correctness of the corpus.

[0043] It should also be noted that the corresponding interface can be a functional entry point in the model under test used to receive external input and return a response. Each interface has a unique interface identifier and interface configuration information. The baseline identifier can be a label used to mark whether the corpus can be used as a benchmark for subsequent testing, such as a boolean value or a specific string. The corpus marked as a baseline indicates that the corpus has been verified and can be shared and used between different projects.

[0044] Specifically, to facilitate understanding of the actual process of corpus generation, refer to Figure 2 , Figure 2 This is a schematic diagram illustrating the corpus generation process provided in Embodiment 1 of this application. During model testing, the device first selects a comparison method, which can be categorized into two paths: intelligent comparison and automated comparison.

[0045] If automated comparison is selected, the device can download and read the pre-stored corpus file based on the uploaded attachment path or attachment ID. If intelligent comparison is selected, the device first obtains the test prompt content input by the user, then calls the AI ​​intelligent factory to arrange and generate a test corpus set through multi-threading. The generated test corpus set is then compiled and uploaded to the comparison platform, where it is then read.

[0046] After the corpus is read, the prompt function is called to generate case names to update the corpus table. Next, the corpus content of the test corpus is written to the table. Finally, it is determined whether to perform a comparison: if so, the comparison execution phase begins; if not, the process ends.

[0047] In this embodiment, after extracting test prompt words from the test instructions, the device calls a large language model to construct a corpus of the test prompt words and receives the raw corpus data returned by the large language model. Then, each piece of corpus data in the raw corpus data is compared with the verification rules stored in the preset knowledge graph to check whether each piece of corpus meets the integrity requirements, covers diverse expressions, contains boundary conditions, and conforms to domain adaptability. Corpus data that fails verification can be removed or marked as invalid. For the raw corpus data that passes verification, each piece of corpus data in the verified raw corpus data is associated with the corresponding interface of the model to be tested, and a baseline identifier is generated for each piece of corpus data. Finally, based on the baseline identifier and the verified raw corpus data, the test corpus set of the model to be tested is determined. Thus, through the knowledge graph-driven quality verification mechanism and the associated reuse of baseline identifiers, the workload of manually verifying corpus quality and repeatedly writing corpora for different projects is reduced, improving the project reuse efficiency of corpus construction.

[0048] Step S30: Obtain the expected results corresponding to the test corpus from the baseline corpus, wherein the baseline corpus includes the mapping relationship between each corpus and the preset standard answer.

[0049] Understandably, the baseline corpus can be a pre-built and stored database or data storage structure used to store validated corpora and their corresponding standard answers. Each corpus entry in this baseline corpus has undergone quality verification and is marked with a baseline identifier, supporting reuse across different testing tasks and projects. The preset standard answers can be pre-annotated standard output data for each corpus entry.

[0050] It is also understandable that the expected result could be the standard output content obtained from the baseline corpus as a comparison benchmark before the test corpus is input into the model to be tested. This expected result is used to compare with the actual output of the model to be tested in order to judge the model performance.

[0051] Step S40: Call the large language model to analyze the expected results and the model output results to obtain the test results of the model to be tested.

[0052] It should be noted that the test results can be comprehensive evaluation information that characterizes the performance of the model under test in this test. The test results may include the difference analysis between the model output results and the expected results, the performance index values, and targeted optimization suggestions. This embodiment does not limit this.

[0053] In practical use, after obtaining the test corpus, the device can iterate through each piece of text in the corpus, using its content or identifier as the query key to retrieve a pre-defined standard answer that maps to the corresponding text in the baseline corpus. This standard answer serves as the expected result for each text match. The expected result and the model output are then sent as input parameters to a large language model for comparative analysis. This analysis identifies the differences between the model output and the expected result, yielding the test result.

[0054] Furthermore, in order to obtain the above test results, in this embodiment, the step of calling the large language model to analyze the expected results and the model output results to obtain the test results of the model to be tested includes: Step S41: Compare the expected result with the model output result to obtain the differential result, and determine the differential corpus corresponding to the differential result.

[0055] It should be noted that the discrepancy results can be the inconsistencies identified after comparing the expected results with the model output results field by field or semantically. These discrepancy results may include types of discrepancies such as missing fields, mismatched field values, extra output fields, or format errors; this embodiment does not impose any limitations on these types.

[0056] It should also be noted that the differentiated corpus can be selected from the test corpus, and the model output results differ from the expected results.

[0057] Step S42: Call the large language model to perform root cause analysis on the expected results and corresponding model output results of the differentiated corpus, and obtain the failure root cause of each differentiated corpus.

[0058] Step S43: Use the failure root causes and the differentiated corpus as the test results of the model to be tested.

[0059] Understandably, the root cause of failure can be a descriptive text generated after root cause analysis, used to explain why specific differential corpora produce output differences.

[0060] For example, suppose the test corpus contains the sentence "Help me book a flight from Beijing to Shanghai". The expected result is "Intent: Book a flight, Departure: Beijing, Destination: Shanghai", while the model output is "Intent: Book a flight, Departure: Beijing". The device compares the two and finds that the model output is missing the "Destination: Shanghai" field, resulting in a discrepancy result of "Destination field missing". The device calls a large language model for root cause analysis, sending the corpus content, expected result, and model output to the large language model. The large language model returns the root cause of the failure: "When processing the destination field, the model failed to correctly extract the slot because 'Shanghai' in the corpus is located at the end of the sentence and is not explicitly marked as the destination."

[0061] In this embodiment, after acquiring the expected results and model output results, the device iterates through each piece of text in the test corpus, comparing the model output result of each piece of text with the corresponding expected result to identify the differences. Next, it identifies the differential text corresponding to the differential results from the test corpus and uses a large language model to perform root cause analysis on the expected results and corresponding model output results of the differential text. Finally, it collects all failure root causes and associates each differential text with its corresponding failure root cause to form a problem location list, which serves as the test result for the model under test. Thus, through automated root cause analysis driven by a large language model, the time cost of manually checking the causes of each piece of text that failed is reduced, improving the interpretability of the test results and the efficiency of problem location.

[0062] This application provides a model testing method. First, test prompt words corresponding to the model to be tested are extracted from the user's test instructions. Then, a test corpus set corresponding to the test prompt words is constructed using a preset knowledge graph. Next, the expected results corresponding to the test corpus set are obtained from a baseline corpus. Finally, a large language model is called to analyze the expected results and the model output. Compared to existing testing methods that rely on manually written corpora and whose corpora are difficult to reuse across projects, this embodiment can automatically generate a test corpus set that can be associated with the baseline corpus by driving a preset knowledge graph through test prompt words. This not only reduces the workload of manually writing corpora but also improves the reusability of corpora between different model projects, thereby improving the overall efficiency of the model testing process.

[0063] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to the above embodiment can be referred to the above description, and will not be repeated hereafter. On this basis, a second embodiment of the dialogue method of this application is proposed, please refer to... Figure 3 , Figure 3 This is a flowchart illustrating Embodiment 2 of the model testing method of this application. To obtain the above model output results, as shown... Figure 3As shown, in this embodiment, the step of inputting the test corpus into the model to be tested and obtaining the model output includes: Step S21: Classify the test corpus set to obtain at least two target corpus sets, and determine the preset test rounds based on the target corpus sets.

[0064] It should be noted that the target corpus can be at least two subsets obtained by dividing the test corpus according to a preset classification rule, and each subset corresponds to a test type.

[0065] For example, simple scenario corpora and complex scenario corpora can be obtained by classifying them according to corpus complexity, or functional verification corpora and boundary condition corpora can be obtained by classifying them according to testing dimensions.

[0066] It should also be noted that the preset test rounds can be determined before the test begins. These rounds specify the execution order and number of times the model under test needs to process different target corpora. Each target corpus can have its own test round, or multiple target corpora can be combined into one test round.

[0067] Step S22: Construct a request body based on the target corpus set of the current test round, and send the request body to the model to be tested to obtain the current output result.

[0068] Step S23: Determine whether the current test round has reached the preset test round.

[0069] Step S24: If the current test round has not reached the preset test round, return to the step of constructing the request body based on the target corpus of the current test round, until the current test round reaches the preset test round, and obtain the model output result.

[0070] Understandably, the request body can be a data structure that encapsulates the content of the target corpus and is organized according to the interface specification of the model to be tested. This request body is used to initiate a call request to the model to be tested.

[0071] It should be understood that the current output can be the response data returned by the model under test after the request body constructed in the current test round is sent to the model under test. By classifying and iterating through multiple rounds of testing, the sufficiency of model validation in complex scenarios can be improved. Complex scenarios can include multi-slot, nested slot, ambiguous expressions, grammatical errors (such as disordered word order, missing components, redundant components, mismatched collocations, logical contradictions, and vague expressions), and generalized expressions (such as recognition generalization and implicit intent).

[0072] In practical use, after obtaining the test corpus, the device classifies it into at least two target corpus sets. After classification, a preset test round is determined based on the resulting target corpus sets. Next, the target corpus set corresponding to the current test round is obtained, and a request body is constructed based on the content of that corpus. The constructed request body is then sent to the model under test, and the response content returned by the model is received and used as the current output. It is then determined whether the current test round has reached the preset test round. If not, it indicates that there are still unexecuted test rounds. The next test round is then initiated, the next target corpus set is obtained, and the above operation is repeated. This process continues until the current test round reaches the preset test round, at which point all target corpus sets have been tested, and a complete model output result is obtained. Thus, through multi-round iterative testing, the potential for missed scenario coverage in a single full-scale test is reduced, improving the sufficiency of model validation in complex scenarios.

[0073] Furthermore, in order to construct the aforementioned request body, in this embodiment, the step of constructing the request body based on the target corpus set of the current test round and sending the request body to the model under test to obtain the current output result includes: Step S221: Obtain the baseline identifier of the target corpus set in the current test round, and determine the interface configuration information and request body template corresponding to the model to be tested based on the baseline identifier.

[0074] It should be noted that the interface configuration information can be a set of parameter configurations required by the interface of the model under test during the communication process. The interface configuration information may include the interface's Uniform Resource Locator, request method type, request header format, authentication method, and timeout setting, etc. This embodiment does not impose any restrictions on this.

[0075] It should also be noted that the request body template can be a predefined structured framework for constructing request data. The request body template may contain placeholders that need to be filled by the corpus content and fixed static fields.

[0076] Step S222: Determine the placeholder positions of the request parameters in the request body template based on the interface configuration information.

[0077] It is understood that the placeholder position can be a specific coordinate or identifier marked in the request body template to indicate the insertion position of the corpus content. The placeholder position can be a string index, field path or special marker symbol, etc., and this embodiment does not limit it.

[0078] Step S223: Obtain the corpus content of each corpus in the target corpus set, and fill the corpus content into the request body template based on the placeholder position to obtain at least two completed request bodies.

[0079] Step S224: Send each request body to the model under test for response using a multi-threaded asynchronous method to obtain the current output result.

[0080] It should be understood that multi-threaded asynchronous mode can be a concurrent processing mode that executes multiple request sending tasks in parallel. In this mode, the device can create multiple threads simultaneously, each thread independently responsible for sending and receiving a request body, and the threads do not block each other, thereby reducing the overall waiting time.

[0081] In practical use, after entering the current test round, the device can obtain the baseline identifier of the target corpus corresponding to the current test round. Then, based on this baseline identifier, it searches the system storage for the interface configuration information and request body template corresponding to the model under test. Next, based on the obtained interface configuration information, it determines the placeholder positions for the request parameters in the request body template. Then, for each corpus in the target corpus, based on the determined placeholder positions, it fills the corresponding placeholder positions in the request body template with the content of each corpus. This filling operation is repeated to generate an independent request body for each corpus in the target corpus, thus obtaining at least two completed request bodies. Finally, a thread pool is created, and an independent thread is allocated to each request body. Each thread is responsible for sending a request body to the interface of the model under test and waiting to receive the returned response data. All threads execute concurrently, collecting the response content returned by each thread, and summarizing these response contents to obtain the current output result. By automatically associating interface configurations driven by baseline identifiers, automatically filling placeholders, and handling multi-threaded asynchronous concurrent requests, the time overhead of manually searching for interface configurations, manually concatenating request bodies, and sending requests serially is reduced, thereby improving test execution efficiency and concurrent processing capabilities.

[0082] For a clearer understanding of the above process, please refer to [link / reference]. Figure 4 , Figure 4 This is a flowchart illustrating the model response provided in Embodiment 2 of this application. First, the device identifies lean projects from the historical testing process, associates them with execution rounds, and saves the relationships between lean projects to an association table. Then, it pushes lean generated cases, saves the lean case ID, and initializes its status to pending. Next, it obtains the configuration of the interface under test and saves a snapshot to support historical traceability. Then, it creates an asynchronous comparison task, simultaneously creating new instances to prevent interference between parallel tasks. This mechanism supports parallel comparison by multiple users or a single user with multiple tasks.

[0083] After completing the above settings, the comparison process is executed. The device first extracts the corpus and processes each request based on the extracted corpus. During processing, placeholders within the request body are replaced to construct a complete request body. The complete request body is then sent to the model's tested interface for processing, and the model's interface response is obtained and processed. During response processing, the obtained standard answer undergoes data format processing, and the formatted standard answer is compared with the response fields to ensure consistency. Finally, the comparison results are saved, completing this comparison task. After completing this comparison task, the above process is repeated serially and iteratively based on the amount of corpus data.

[0084] Based on the first and / or second embodiments of this application, in the third embodiment of this application, the content that is the same as or similar to that in embodiments one and two above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 5 , Figure 5 This is a flowchart illustrating Embodiment 3 of the model testing method of this application. To obtain further analysis of the above test results, such as... Figure 5 As shown, in this embodiment, after the step of calling the large language model to analyze the expected result and the model output result to obtain the test result of the model to be tested, the method further includes: Step S51: Extract the performance metrics of the model under test in multiple historical test rounds.

[0085] It should be noted that performance metrics can be numerical parameters used to quantitatively evaluate the performance of the model under test during the test, such as average response time, accuracy, precision, recall, F1 score, and success rate. This embodiment does not impose any restrictions on these metrics.

[0086] Among them, accuracy is used to measure overall correctness; precision is used to assess the accuracy of positive predictions; recall is used to reflect the coverage of the actual situation; and F1 score is used as a comprehensive balanced indicator.

[0087] It should also be noted that historical test rounds can be any test rounds that have been completed before the current test. Each historical test round corresponds to a complete test execution process and records the performance index values ​​of the model under test in that round.

[0088] Step S52: Call the large language model to analyze the time-series changes of the performance indicators and obtain the performance change trend.

[0089] Understandably, the performance change trend can be a description of the change pattern output by the large language model after analyzing the time-series changes of performance indicators, such as continuous rise, continuous fall, fall followed by rise, rise followed by fall, low level remain unchanged, or high level remain unchanged. This embodiment does not limit this.

[0090] Step S53: Determine the corpus supplementation strategy for the test corpus based on the root causes of failure and the performance change trend, and determine the model iteration strategy for the model to be tested based on the performance change trend.

[0091] Step S54: Generate a strategy optimization report corresponding to the test results based on the corpus supplementation strategy and the model iteration strategy.

[0092] It is also understood that the corpus supplementation strategy can be a plan and implementation scheme for adding new corpus to address the insufficient coverage or quality defects of the current test corpus. The corpus supplementation strategy may include the type of corpus to be supplemented, the quantity of corpus to be supplemented, the generation method, and the test scenarios to be covered.

[0093] It should be understood that a model iteration strategy can be an improvement plan for the model under test, based on performance change trends and root causes of failure. This strategy may include hyperparameter tuning, training data supplementation, API call chain investigation, or model architecture adjustment.

[0094] In practical use, after obtaining the root causes of failures and the model output results, the device extracts the performance metrics of the model under test across multiple historical test rounds. Then, it uses a large language model to analyze the temporal changes of these performance metrics, identifying rising, falling, fluctuating, or flat trends as performance change trends. Subsequently, it combines the types of missing corpus coverage reflected in the root causes of failures with the model weaknesses indicated by the performance change trends to determine which test scenarios the current test corpus is insufficient in. Based on this, it formulates the types, quantities, and generation methods of the corpus to be added, forming a corpus supplementation strategy. Simultaneously, it identifies potential bottlenecks in the model based on the performance change trends and develops corresponding model improvement plans, such as adjusting hyperparameters, supplementing training data, or troubleshooting interface links. Finally, based on the corpus supplementation strategy and the model iteration strategy, it generates a strategy optimization report corresponding to the test results. Thus, through historical trend analysis and root cause fusion driven by a large language model, the subjectivity and time cost of manually analyzing performance fluctuations and formulating optimization strategies are reduced.

[0095] Furthermore, in order to obtain the above-mentioned model iteration strategy, in this embodiment, the step of determining the model iteration strategy of the model to be tested based on the performance change trend includes: Step S531: Obtain the test context information of the model to be tested in the historical test rounds.

[0096] It should be noted that the test context information can be a set of environment configurations and auxiliary data related to the test execution of the model under test in historical test rounds. This test context information may include test execution time, test environment configuration parameters, version number of the model under test, network status of the interface call chain, number of concurrent requests, and system resource usage, etc.

[0097] Step S532: Determine the time-series change trajectory of the performance index in the historical test rounds based on the performance change trend.

[0098] It should also be noted that the time-series change trajectory can be a description of the change path of performance indicators in historical test rounds, connected in chronological order.

[0099] Step S533: Based on the root cause of failure and the time-series change trajectory, call the large language model to generate an optimization action set, which includes a model hyperparameter adjustment strategy, a training data supplementation strategy, and an interface call chain investigation strategy.

[0100] Step S534: Generate the model iteration strategy for the model to be tested based on the model hyperparameter adjustment strategy, the training data supplementation strategy, and the interface call chain investigation strategy.

[0101] Understandably, an optimization action set can be a collection of multiple specific optimization actions, each corresponding to an improvement operation for the model or testing process. This optimization action set includes three types of actions: model hyperparameter tuning strategies, training data supplementation strategies, and API call chain investigation strategies.

[0102] Among these, the hyperparameter tuning strategy can be a plan to modify the internal parameters of the model under test. This strategy includes the names of the hyperparameters to be adjusted, their adjusted values, and the basis for the adjustment. The training data supplementation strategy can be a plan to add extra training samples to the model under test. This strategy can include the data type, data source, data volume, and annotation method to be supplemented. The interface call chain troubleshooting strategy can be a plan to diagnose and optimize the interface call process on which the model under test depends.

[0103] In this embodiment, the device further enhances the comprehensiveness of the model iteration strategy by driving optimization through a large language model that integrates test context, time sequence trajectory and failure root cause.

[0104] Furthermore, in another feasible embodiment, in addition to generating the aforementioned strategy optimization report, this embodiment will also interpret the report, referring to... Figure 6 , Figure 6This is a flowchart illustrating the report interpretation process provided in Embodiment 3 of this application. After the comparison is completed, various indicators are calculated to generate the aforementioned strategy optimization report. Next, the report link is processed and a message notification is sent. Simultaneously, the report content is sent for AI interpretation. The AI ​​interpretation scenarios can cover the following four core scenarios: The first step is a stability analysis of the comparison results. This involves analyzing the distribution of result types, performance, and test case response times for this request. Specific cases that exceed expected metrics will be listed to facilitate user analysis and identification.

[0105] The second is corpus integrity analysis. Through intelligent factory orchestration and AI testing strategies, it can check whether the corpus covers dimensions such as diverse expressions (synonyms, different sentence structures, referential resolution, multilingualism, etc.), complex scenarios (such as multi-slot or intent extraction, nesting, ambiguity, grammatical errors, generalized expressions, etc.), boundary conditions and extreme cases (such as input length, noise errors, missing redundant information, etc.), and security and reliability (such as content security, privacy protection, etc.), and provide prompts for any areas not covered.

[0106] The third step is the accuracy analysis of the corpus and results. An accuracy analysis is performed on the standard answers, actual answers, and the corpus to identify existing problems, analyze possible causes, and provide optimization suggestions.

[0107] The fourth step is historical baseline comparison analysis. Through smart factory orchestration, the test results of the interface's historical baseline are compared and checked with the current test results, the change curves are analyzed, and corresponding optimization suggestions are given based on trends (such as first decreasing and then increasing, continuous decreasing, continuous increasing, etc.).

[0108] Based on the AI ​​interpretation described above, a multi-dimensional intelligent analysis report can be generated, including stability, completeness, accuracy, and historical trend comparisons. After generating the multi-dimensional intelligent analysis report, visualization charts (such as failure type pie charts, F1 index laser charts, and response time trend charts) are also output based on the report to enrich the model testing results.

[0109] It should be noted that the above examples are for understanding this application only and do not constitute a limitation on the model testing method of this application. Any simple modifications based on this technical concept are within the scope of protection of this application. All actions involving the acquisition of signals, information, or data in this application are performed in accordance with the relevant data protection regulations of the country where the application is located and with authorization from the owner of the relevant device.

[0110] This application also provides a model testing device, please refer to... Figure 7 , Figure 7This is a block diagram of the module structure of the model testing device according to an embodiment of this application; in this embodiment, the model testing device includes: The prompt word module 701 is used to extract the test prompt words corresponding to the model to be tested from the user's test instructions; Corpus module 702 is used to construct a test corpus set corresponding to the test prompt words through a preset knowledge graph, and input the test corpus set into the model to be tested to obtain the model output result; The expected module 703 is used to obtain the expected results corresponding to the test corpus from the baseline corpus, wherein the baseline corpus includes the mapping relationship between each corpus and the preset standard answer; Analysis module 704 is used to call the large language model to analyze the expected results and the model output results to obtain the test results of the model to be tested.

[0111] This embodiment first extracts test prompt words corresponding to the model to be tested from the user's test instructions, and constructs a test corpus set corresponding to the test prompt words through a preset knowledge graph. Then, it obtains the expected results corresponding to the test corpus set from the baseline corpus, and calls a large language model to analyze the expected results and the model output results. Compared with the existing testing methods that rely on manually written corpora and whose corpora are difficult to reuse across projects, this embodiment can automatically generate a test corpus set that can be associated with the baseline corpus by driving a preset knowledge graph through test prompt words. This not only reduces the workload of manually writing corpora, but also improves the reuse rate of corpora between different model projects, thereby improving the overall efficiency of the model testing process.

[0112] In one implementation, the corpus module 702 is further configured to call a large language model to construct a corpus of the test prompt words, generating raw corpus data; perform quality verification on the raw corpus data according to a preset knowledge graph; associate each piece of corpus data in the verified raw corpus data with the corresponding interface of the model to be tested to obtain a baseline identifier; and determine the test corpus set of the model to be tested based on the baseline identifier and the verified raw corpus data.

[0113] In one implementation, the corpus module 702 is further configured to classify the test corpus set to obtain at least two target corpus sets, and determine a preset test round based on the target corpus sets; construct a request body based on the target corpus set of the current test round, and send the request body to the model to be tested to obtain the current output result; determine whether the current test round has reached the preset test round; if the current test round has not reached the preset test round, return to the step of constructing a request body based on the target corpus set of the current test round, until the current test round reaches the preset test round, and obtain the model output result.

[0114] In one implementation, the corpus module 702 is further configured to obtain the baseline identifier of the target corpus set in the current test round, and determine the interface configuration information and request body template corresponding to the model under test based on the baseline identifier; determine the placeholder positions of the request parameters in the request body template based on the interface configuration information; obtain the corpus content of each corpus in the target corpus set, and fill the corpus content into the request body template based on the placeholder positions to obtain at least two completed request bodies; and send each request body to the model under test for response using a multi-threaded asynchronous method to obtain the current output result.

[0115] In one implementation, the analysis module 704 is further configured to compare the expected results with the model output results to obtain differential results, and determine the differential corpus corresponding to the differential results; call a large language model to perform root cause analysis on the expected results and the corresponding model output results of the differential corpus to obtain the failure root cause of each differential corpus; and use the failure root cause and the differential corpus as the test results of the model to be tested.

[0116] In one implementation, the analysis module 704 is further configured to extract performance metrics of the model under test in multiple historical test rounds; call the large language model to analyze the temporal changes of the performance metrics to obtain performance change trends; determine the corpus supplementation strategy for the test corpus based on the root causes of failure and the performance change trends, and determine the model iteration strategy for the model under test based on the performance change trends; and generate a strategy optimization report corresponding to the test results based on the corpus supplementation strategy and the model iteration strategy.

[0117] In one implementation, the analysis module 704 is further configured to: acquire test context information of the model under test in the historical test rounds; determine the temporal change trajectory of the performance index in the historical test rounds based on the performance change trend; generate an optimization action set by calling the large language model based on the failure root cause and the temporal change trajectory, the optimization action set including a model hyperparameter adjustment strategy, a training data supplementation strategy, and an interface call link investigation strategy; and generate a model iteration strategy for the model under test based on the model hyperparameter adjustment strategy, the training data supplementation strategy, and the interface call link investigation strategy.

[0118] Other embodiments or specific implementations of the model testing device of this application can be found in the above-described method embodiments, and will not be repeated here.

[0119] The model testing apparatus provided in this application, employing the model testing method described in the above embodiments, can solve the technical problem of low efficiency in existing model testing processes. Compared with the prior art, the beneficial effects of the model testing apparatus provided in this application are the same as those of the model testing method provided in the above embodiments, and other technical features in the model testing apparatus are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0120] This application provides a model testing device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the model testing methods in the above embodiments.

[0121] The following is for reference. Figure 8 , Figure 8 This is a schematic diagram of the hardware operating environment involved in the model testing device in the embodiments of this application, showing a structural schematic diagram suitable for implementing the model testing device in the embodiments of this application. The model testing device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (PADs), portable media players (PMPs), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 8 The model testing equipment shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0122] like Figure 8As shown, the model testing device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the model testing device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the model test equipment to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows a model test equipment with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0123] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0124] The model testing equipment provided in this application, employing the model testing method described in the above embodiments, can solve the technical problem of low efficiency in existing model testing processes. Compared with the prior art, the beneficial effects of the model testing equipment provided in this application are the same as those of the model testing method provided in the above embodiments, and other technical features of the model testing equipment are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0125] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0126] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0127] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the model testing method in the above embodiments.

[0128] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, radio frequency (RF), etc., or any suitable combination thereof.

[0129] The aforementioned computer-readable storage medium may be included in the model testing equipment; or it may exist independently and not be assembled into the model testing equipment.

[0130] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by the model testing device, the model testing device performs the following actions: extracts test prompt words corresponding to the model to be tested from the user's test instructions; constructs a test corpus set corresponding to the test prompt words through a preset knowledge graph, and inputs the test corpus set into the model to be tested to obtain the model output result; obtains the expected result corresponding to the test corpus set from a baseline corpus, the baseline corpus including the mapping relationship between each corpus and a preset standard answer; and calls a large language model to analyze the expected result and the model output result to obtain the test result of the model to be tested.

[0131] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0132] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation that may be implemented in systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0133] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0134] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described model testing method, thereby solving the technical problem of low efficiency in existing model testing processes. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the model testing method provided in the above embodiments, and will not be repeated here.

[0135] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the model testing method described above.

[0136] The computer program product provided in this application can solve the technical problem of low efficiency in existing model testing processes. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the model testing method provided in the above embodiments, and will not be repeated here.

[0137] The above description is only a part of the embodiments of this application and does not limit the scope of protection of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included in the scope of protection of this application.

Claims

1. A model testing method, characterized in that, The model testing method includes: Extract test prompts corresponding to the model to be tested from the user's test instructions; A test corpus corresponding to the test prompt words is constructed by using a pre-defined knowledge graph, and the test corpus is input into the model to be tested to obtain the model output results; Obtain the expected results corresponding to the test corpus from the baseline corpus, which includes the mapping relationship between each corpus and the preset standard answer; The large language model is invoked to analyze the expected results and the output results of the model, and the test results of the model to be tested are obtained.

2. The method as described in claim 1, characterized in that, The step of constructing the test corpus corresponding to the test prompt words through a preset knowledge graph includes: The large language model is invoked to construct a corpus of the test prompt words, generating raw corpus data. The quality of the original corpus data is verified based on a preset knowledge graph. Each piece of text in the original text data that has passed the verification is associated with the corresponding interface of the model to be tested to obtain a baseline identifier; Based on the baseline identifier and the original corpus data after verification, the test corpus set of the model to be tested is determined.

3. The method as described in claim 1, characterized in that, The step of inputting the test corpus into the model to be tested and obtaining the model output includes: The test corpus is classified to obtain at least two target corpus sets, and a preset test round is determined based on the target corpus sets. A request body is constructed based on the target corpus set of the current test round, and the request body is sent to the model to be tested to obtain the current output result; Determine whether the current test round has reached the preset test round; If the current test round has not reached the preset test round, return to the step of constructing the request body based on the target corpus set of the current test round, until the current test round reaches the preset test round, and obtain the model output result.

4. The method as described in claim 3, characterized in that, The step of constructing a request body based on the target corpus set of the current test round, sending the request body to the model under test, and obtaining the current output result includes: Obtain the baseline identifier of the target corpus set in the current test round, and determine the interface configuration information and request body template corresponding to the model to be tested based on the baseline identifier; The placeholder positions for request parameters in the request body template are determined based on the interface configuration information. Obtain the content of each corpus in the target corpus, and fill the corpus content into the request body template based on the placeholder position to obtain at least two completed request bodies; Each request body is sent to the model under test in an asynchronous, multi-threaded manner to obtain the current output result.

5. The method as described in claim 1, characterized in that, The step of calling the large language model to analyze the expected results and the model output to obtain the test results of the model under test includes: The expected results are compared with the model output to obtain the differential results, and the differential corpus corresponding to the differential results is determined. The large language model is invoked to perform root cause analysis on the expected results and corresponding model output results of the differentiated corpus, so as to obtain the root cause of failure for each of the differentiated corpus entries; The root causes of failure and the differentiated corpus are used as the test results for the model under test.

6. The method as described in claim 5, characterized in that, After the step of calling the large language model to analyze the expected results and the model output to obtain the test results of the model to be tested, the method further includes: Extract the performance metrics of the model under test in multiple historical test rounds; The large language model is invoked to analyze the time-series changes of the performance indicators and obtain the performance change trend; Based on the root causes of failure and the performance change trend, a corpus supplementation strategy for the test corpus is determined, and a model iteration strategy for the model to be tested is determined based on the performance change trend. Based on the corpus supplementation strategy and the model iteration strategy, a strategy optimization report corresponding to the test results is generated.

7. The method as described in claim 6, characterized in that, The step of determining the model iteration strategy for the model under test based on the performance change trend includes: Obtain the test context information of the model to be tested in the historical test rounds; Based on the performance change trend, determine the time-series change trajectory of the performance index in the historical test rounds; Based on the root cause of failure and the time-series change trajectory, the large language model is invoked to generate an optimization action set, which includes a model hyperparameter adjustment strategy, a training data supplementation strategy, and an interface call chain investigation strategy. Based on the model hyperparameter adjustment strategy, the training data supplementation strategy, and the interface call chain investigation strategy, a model iteration strategy for the model to be tested is generated.

8. A model testing device, characterized in that, The device includes: The prompt word module is used to extract the test prompt words corresponding to the model to be tested from the user's test instructions; The corpus module is used to construct a test corpus set corresponding to the test prompt words through a preset knowledge graph, and input the test corpus set into the model to be tested to obtain the model output results; The expected module is used to obtain the expected results corresponding to the test corpus from the baseline corpus, which includes the mapping relationship between each corpus and the preset standard answer; The analysis module is used to call the large language model to analyze the expected results and the model output results to obtain the test results of the model to be tested.

9. A model testing device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the model testing method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the model testing method as described in any one of claims 1 to 7.

Images