Generation method and execution device for test case of large language model

Abstract

The invention provides a generation method and an execution device for a test case of a large language model. The method comprises the following steps: generating a test set based on a preset rule; the test set comprises a plurality of use case sets, and the test set is matched with a pre-configured interface; and the test set tests a plurality of tasks in the test tasks through the matched interfaces to obtain a test result. According to the method, the related performance data can be collected while the function case is executed, the performance test is completed while the function is tested, the test time can be saved, and the test cost can be reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing, and in particular to a method for generating test cases for a large language model and an execution device. Background Art

[0002] In the large language model testing method, the current testing method is usually to manually prepare test cases, then execute the interface, and view the results to achieve the purpose of verifying the function. However, this method currently has three links with low efficiency and there is much room for improvement.

[0003] In the use case preparation stage, test cases need to be collected and designed. In the process of large model testing, the types of test cases need to be considered, such as Frequently Asked Questions (FAQ), intent identification, root cause analysis, indicator search, context understanding, weather, customers, etc. There is also a lot of data to be prepared. Test cases with the same intent but different questions will take up a lot of time for testers in the test case preparation stage. When testing the same interface, it is basically single-threaded. Since the number of use cases in the test of large language models ranges from a few hundred to millions, conventional single-task single-concurrency will also consume a lot of time to execute the test, which will bring the risk of delay and bug launch to the project launch. After the test is completed, the generated test report only focuses on the function of the use case, showing the execution results of the use case, and the differences between the expected results and the actual results. Due to the traditional interface test, the number of use cases is not particularly large, and the performance of the interface execution is not paid attention to, resulting in the need for additional development of the interface performance, which not only takes a long time and is costly, but also causes the risk of project delay or performance issues. Summary of the invention

[0004] The present disclosure provides a method for generating test cases for a large language model and an execution device to at least solve the above technical problems existing in the prior art.

[0005] According to a first aspect of the present disclosure, a method for generating test cases for a large language model is provided, the method comprising:

[0006] Generate a test set based on preset rules; the test set includes multiple use case sets with corresponding functions, and each use case set includes multiple test cases;

[0007] Matching the test set with a preconfigured interface according to the test task;

[0008] The test set tests multiple tasks in the test task through the matching interface to obtain the test results.

[0009] In one possible implementation, generating a test set based on preset rules includes:

[0010] Match the corresponding use case set initial template according to the required use case set type;

[0011] Determine dimension parameters according to the test scenario, and add the dimension parameters to the initial template of the use case set to obtain a first template of the use case set;

[0012] The first template of the use case set is tested using random entity data. If a target use case is obtained, the first template of the use case set is used as the target template of the use case set; otherwise, the dimension parameters are modified to update the initial template of the use case set until the target use case is obtained when the entity data is input, and the target template of the use case set is output;

[0013] Generate test cases using the use case set target template to obtain a use case set with multiple attributes;

[0014] The test set is obtained based on a set of use cases with multiple attributes.

[0015] In one possible implementation, the interface includes multiple interfaces, and the method of configuring the interface includes:

[0016] Set the interface name, interface address, interface request parameters, header information, interface checksum field, and field path of each interface.

[0017] In one possible implementation, matching the test set with a preconfigured interface includes:

[0018] Set the task name, corresponding interface, number of concurrent tasks corresponding to the test set, set the filtering method of the test case set in the test set, and set the scheduled task;

[0019] Among them, each task matches multiple test sets.

[0020] In one possible implementation, the screening method of the test case set in the test set includes at least one of the following:

[0021] Divide the test set into batches according to the specified batch size, and then obtain a percentage of data from each batch;

[0022] Query all use cases through SQL, sort by use case address, and filter use cases by index range.

[0023] In one possible implementation, the test result includes:

[0024] Use case progress, success rate, and results of executed use cases.

[0025] In one possible implementation, the dimension parameters include: dimension data, use case type, generation rules, target answers, and use case uploads.

[0026] In one possible implementation, it also includes:

[0027] Based on different tasks, the use case set is operated; the operation on the use case set includes:

[0028] Edit, delete, view, download, and retry the use case set if failed.

[0029] According to a second aspect of the present disclosure, a device for generating test cases for a large language model is provided, the device comprising:

[0030] A generation module is used to generate a test set based on preset rules; the test set includes multiple use case sets

[0031] A matching module, matching the test set with a preconfigured interface;

[0032] The test module is used for the test set to test multiple tasks in the test task through the matching interface to obtain the test results.

[0033] In one possible implementation, the generating module includes:

[0034] A matching unit, used to match a corresponding use case set initial template according to the required use case set type;

[0035] A determination unit, configured to determine a dimension parameter according to a test scenario, and add the dimension parameter to the initial template of the use case set to obtain a first template of the use case set;

[0036] An output unit is used to test the first template of the use case set using random entity data, and if a target use case is obtained, the first template of the use case set is used as the target template of the use case set; otherwise, the dimension parameters are modified to update the initial template of the use case set until the target use case is obtained when the entity data is input, and the target template of the use case set is output;

[0037] A generating unit, used to generate test cases using the use case set target template to obtain a use case set with multiple attributes;

[0038] The acquisition unit is used to obtain a test set based on a set of use cases with multiple attributes.

[0039] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0040] at least one processor; and

[0041] a memory communicatively connected to the at least one processor; wherein,

[0042] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the present disclosure.

[0043] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method described in the present disclosure.

[0044] The test case generation method and execution device for a large language model disclosed in the present invention can collect relevant performance of the test case while executing the test task, and complete the performance test while testing the function, which not only saves the test time but also reduces the test cost.

[0045] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, in which:

[0047] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts.

[0048] Figure 1 The following is a schematic diagram showing the implementation process of the method for generating test cases for a large language model according to an embodiment of the present disclosure. Figure 1 ;

[0049] Figure 2 The following is a schematic diagram showing the implementation process of the method for generating test cases for a large language model according to an embodiment of the present disclosure. Figure 2 ;

[0050] Figure 3 A schematic diagram of the structure of an execution device for a test case of a large language model according to an embodiment of the present disclosure is shown;

[0051] Figure 4 A schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0052] In order to make the purpose, features, and advantages of the present disclosure more obvious and easy to understand, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0053] The testing phase of a large language model basically relies entirely on manual questioning or interface questions, and then observing the results returned by the model. Among them, use case design, execution, and performance analysis of results under different concurrency are the three most time-consuming links. Large language model testing requires a tool or product that can help testers improve testing efficiency.

[0054] In the use case design phase, a product is needed to help testers quickly generate test cases and reduce the number of incomplete use case designs and design errors. The product should be simple and easy to use, efficient, accurate, and comprehensive; it should be able to solidify and precipitate the use case design experience of testers, and standardize and standardize the use case test design process through system products / tools. In the existing use case design phase, use cases are basically manually uploaded or written online to the system, but such products have the following problems: Thousands of use cases will consume a lot of time when designing. When the use case considers too many questions and dimensions, the designed use cases may not be complete, resulting in reduced coverage. A use case cannot meet both intent recognition testing and function point testing at the same time.

[0055] In the execution phase of use cases, a product is needed to improve the efficiency of testers' use case execution. Under the maximum number of concurrency allowed by the model, the desired number of parallel operations can be entered for parallel execution to reduce the execution time of the use case. Existing use cases are basically executed in a single-thread / single-process manner, which has limited efficiency improvements.

[0056] When performing performance analysis on a large language model, performance testing is usually done separately using additional performance tools. This consumes additional testing resources and does not allow the performance data of the interface to be obtained while executing the use case. It is also impossible to analyze the performance of the interface, resulting in time and personnel costs caused by repeated execution of the use case.

[0057] A method for generating test cases for a large language model and an execution device provided in an embodiment of the present application are described below in conjunction with the accompanying drawings.

[0058] like Figure 1 As shown, the present application provides a method for generating test cases for a large language model, the method comprising:

[0059] S101, generating a test set based on preset rules; the test set includes a plurality of use case sets with corresponding functions, and each use case set includes a plurality of test cases;

[0060] It is understandable that in this application, test cases can be generated through dimensional parameter data. First, an interface is used to determine whether to use a large language and answer verification rules. The use case types are divided into two types: FAQ and intent recognition. Among them, the FAQ type includes question type, channel and time range; intent recognition includes question type, template, synonyms and channel. After the use case executes the interface, the returned content uses content that contains or is equal to the standard answer. Finally, the use case is uploaded to the use case set, and multiple use case sets constitute the test set.

[0061] The test set provided in this application can be edited, deleted, and viewed by users at any time (name, subset name, number of use cases, creator, creation time, and status).

[0062] S102, matching the test set with a preconfigured interface according to the test task;

[0063] It is understandable that the test set in this application contains multiple use case sets. Depending on the execution task, the corresponding interface performance is different. Therefore, according to the different test tasks, the use case sets in the test set are matched with the corresponding interfaces. It is understandable that a task list is provided in this application, in which all tasks can be viewed, and the name of each task, the associated test set, the number of test set cases, the number of task cases, the creator, the executor, the task start time, the task end time, the status, the progress, the number of successes, the number of failures, and the success rate are displayed. Based on the test task, the use case set name is set, the test set is selected, the filtering rules are selected, the interface is selected, the number of concurrency is selected, and the scheduled task is set; among them, when the use case set name is repeated, the use case will be added incrementally based on the original test set, and when a duplicate use case appears, it will be skipped; otherwise, it will be created normally.

[0064] S103, the test set tests multiple tasks in the test task through the matching interface to obtain a test result.

[0065] After the interface matches the use case set in the test set, the task is started and all the use cases that meet the rules are executed. The number of executed cases is the number of task use cases. After the use cases are executed, the failed use cases are re-run. The execution results are merged into the previous execution results. Whether the task is being executed or completed, the test report can be viewed and downloaded. When a test task fails, the task is terminated and all processes of the task are stopped. After the test task is completed, the created task is deleted, and all execution records and test reports are deleted upward, but the associated test set is not deleted.

[0066] The method for generating test cases for a large language model provided in the present application first generates a test set through preset rules, wherein the test set has multiple use case sets, each use case set has corresponding functions, and each use case set includes multiple test cases, for example, 500 test cases. In addition, an interface is pre-set in the present application, and for a test task, a use case set of a corresponding function tests the test task through a matching interface to obtain a test result. It should be noted that the test task includes multiple, different use case sets test the corresponding tasks through corresponding interfaces, so that the test case collects relevant performance at the same time as the test task is executed, and completes the performance test while testing the function, which not only saves the test time, but also reduces the test cost.

[0067] In some embodiments, generating a test set based on preset rules includes:

[0068] Match the corresponding use case set initial template according to the required use case set type;

[0069] Determine dimension parameters according to the test scenario, and add the dimension parameters to the initial template of the use case set to obtain a first template of the use case set;

[0070] The first template of the use case set is tested using random entity data. If a target use case is obtained, the first template of the use case set is used as the target template of the use case set; otherwise, the dimension parameters are modified to update the initial template of the use case set until the target use case is obtained when the entity data is input, and the target template of the use case set is output;

[0071] Generate test cases using the use case set target template to obtain a use case set with multiple attributes;

[0072] The test set is obtained based on a set of use cases with multiple attributes.

[0073] In this application, the dimensional data obtains business data by calling a third-party interface, and splices the words according to the specified dimensional data according to the corresponding template as a use case and answer. The dimensional data sample is: {"org-name":"Shanghai","synonym":["Shanghai District","Shanghai Province","Shanghai"],"index-name":["ccr","Single Ticket Transport Capacity Cost"],"date":["Yesterday","The day before yesterday","The day before yesterday"]}, template case: ["${date}'s${org-name}'s${index-name}"] answer (including relationship): "${date}&${index-name}", the intention will be automatically generated, one of the cases is: {"index-name":"ccr","org-name":"Shanghai","date":"Yesterday"}; according to the template and generation rules, the desired use case can be generated through the operation of Cartesian sets.

[0074] Specifically, Figure 2 As shown, in this application, the existing initial template can be used to fill in the dimension parameters to generate a test set. Specifically, select the type of use case set to be generated, and import the use case set template to be generated from the template list; according to the scenario to be tested, if it is FAQ, select questions, channels, and time; if it is intent recognition: select questions, corpus templates, synonyms, channels, and answers, where corpus templates and answers can have dimension variables, such as: what is ${index} of ${data}. Save as a template to facilitate the generation of the same use case type next time; use an entity data to generate a test data, display it on the page, and let the tester confirm whether it is the use case he wants. If the generated use case is what he wants, he can proceed to the next step, otherwise modify the parameters and repeat the above steps; finally, the use case will obtain all the values ​​corresponding to the dimension parameters through the third-party interface or dimension parameter dictionary to generate the test case. At this time, the state of the test set is generating, and an approximate number of test sets will be estimated. When the generation is completed, the state is generation completed. At this time, the number of test sets will be updated to the actual number of use cases, and the use case generation is completed.

[0075] When the automatic generation of test cases is not supported, you can use Excel to upload the test set.

[0076] The template in this application can be a FAQ template. When generating a FAQ use case, the relevant parameters of the generated use case can be saved as a template. It can also be an intent recognition template. When generating an intent recognition use case, the relevant parameters of the generated use case can be saved as a template.

[0077] In some embodiments, the interface includes multiple interfaces, and the method of configuring the interface includes:

[0078] Set the interface name, interface address, interface request parameters, header information, interface checksum field, and field path of each interface.

[0079] Specifically, in this application, before using the product, the tester needs to configure the interface-related information. The configuration method is to fill in the interface name, interface address, interface request parameters, header information, and interface check field and field path.

[0080] In some embodiments, matching the test set with a preconfigured interface includes:

[0081] Set the task name, corresponding interface, number of concurrent tasks corresponding to the test set, set the filtering method of the test case set in the test set, and set the scheduled task;

[0082] Among them, each task matches multiple test sets.

[0083] Specifically, associate the above interface with the use case set as follows:

[0084] Set the task name, select the interface, set the number of concurrencies, select the test set, set the filtering method and filtering value, and set the scheduled task to complete the task creation. After the use case executes the interface, the returned content contains or is equal to the standard answer; for inclusion verification, you can use "&" and, such as aa&bb: it needs to contain multiple strings, and the use case execution passes if all are included, otherwise it fails; you can also use "|" or, such as aa|bb: if only one of the multiple strings is included, it passes; if there are both & and |, execute & first and then execute or; for equality verification, the response content needs to be strictly equal to the standard answer, and if they are equal, the verification passes, otherwise the use case execution fails.

[0085] The characters in this application can also be scheduled tasks, which can be turned on or off, and can be executed once, multiple times or in a loop. Start the task and execute all use cases that meet the rules. The number of executions is the number of task use cases. After executing the use cases, rerun the failed use cases. The results of the execution are merged into the results of the previous execution. Whether the task is in execution or completed, you can view and download the test report. After the task is completed, delete the created task, and delete all execution records and test reports upwards, but do not delete the associated test set.

[0086] When creating the character in the test task in this application, set the use case set name, select the test set, select the filtering rules, select the interface, select the number of concurrency, and set the scheduled task; among them, when the use case set name is repeated, the use case will be incrementally added based on the original test set. When duplicate use cases appear, they will be skipped; otherwise, it will be created normally.

[0087] In some embodiments, setting a screening method for a test case set in a test set includes at least one of the following:

[0088] Divide the test set into batches according to the specified batch size, and then obtain a percentage of data from each batch;

[0089] Query all use cases through SQL, sort by use case address, and filter use cases by index range.

[0090] Specifically, in this application, a task can set multiple test sets, and each test set has its own use case screening method.

[0091] The number of use cases will show the total number of use cases in each test set after filtering according to various rules. There are two types of filtering rules: one is percentage, which divides the test set into batches according to the specified batch, and then obtains the percentage of data from each batch; the other is interval, which uses SQL to query all use cases, sort by use case id, and filter use cases by index interval.

[0092] In some embodiments, the test results include:

[0093] Use case progress, success rate, and results of executed use cases.

[0094] The dimension parameters include: dimension data, use case type, generation rules, target answer and use case upload.

[0095] like Figure 3 As shown, an embodiment of the present application provides an execution device for a test case of a large language model, the device comprising:

[0096] A generating module 301 is used to generate a test set based on preset rules; the test set includes a plurality of use case sets with corresponding functions, and each use case set includes a plurality of test cases;

[0097] A matching module 302 matches the test set with a preconfigured interface according to a test task;

[0098] The test module 303 is used for the test set to test multiple tasks in the test task through the matching interface to obtain the test result.

[0099] The execution device of the test case for a large language model provided in the present application generates a test set based on preset rules through a generation module 301; the test set includes multiple use case sets with corresponding functions, and each use case set includes multiple test cases; the matching module 302 matches the test set with a preconfigured interface according to the test task; the test module 303 tests the multiple tasks in the test task through the matched interface to obtain the test result.

[0100] In some embodiments, the generating module comprises:

[0101] A matching unit, used to match a corresponding use case set initial template according to the required use case set type;

[0102] A determination unit, configured to determine a dimension parameter according to a test scenario, and add the dimension parameter to the initial template of the use case set to obtain a first template of the use case set;

[0103] An output unit is used to test the first template of the use case set using random entity data, and if a target use case is obtained, the first template of the use case set is used as the target template of the use case set; otherwise, the dimension parameters are modified to update the initial template of the use case set until the target use case is obtained when the entity data is input, and the target template of the use case set is output;

[0104] A generating unit, used to generate test cases using the use case set target template to obtain a use case set with multiple attributes;

[0105] The acquisition unit is used to obtain a test set based on a set of use cases with multiple attributes.

[0106] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.

[0107] Figure 4 A schematic block diagram of an example electronic device 400 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0108] like Figure 4 As shown, the device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the device 400 can also be stored. The computing unit 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0109] A number of components in the device 400 are connected to the I / O interface 405, including: an input unit 406, such as a keyboard, a mouse, etc.; an output unit 407, such as various types of displays, speakers, etc.; a storage unit 408, such as a disk, an optical disk, etc.; and a communication unit 409, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 409 allows the device 400 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0110] The computing unit 401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 401 performs the various methods and processes described above, such as a method for generating test cases for a large language model. For example, in some embodiments, the method for generating test cases for a large language model may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 400 via the ROM 402 and / or the communication unit 409. When the computer program is loaded into the RAM 403 and executed by the computing unit 401, one or more steps of the method for generating test cases for a large language model described above may be performed. Alternatively, in other embodiments, the computing unit 401 may be configured to execute the method for generating test cases for a large language model in any other appropriate manner (for example, by means of firmware).

[0111] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0112] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0113] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0114] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0115] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0116] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0117] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0118] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0119] The above is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A method for generating test cases for a large language model, characterized in that: The method comprises: Generate a test set based on preset rules; the test set includes multiple use case sets with corresponding functions, and each use case set includes multiple test cases; Matching the test set with a preconfigured interface according to the test task; The test set tests multiple tasks in the test task through the matching interface to obtain the test results.

2. The method according to claim 1, characterized in that Generate test sets based on preset rules, including: Match the corresponding use case set initial template according to the required use case set type; Determine dimension parameters according to the test scenario, and add the dimension parameters to the initial template of the use case set to obtain a first template of the use case set; The first template of the use case set is tested using random entity data. If a target use case is obtained, the first template of the use case set is used as the target template of the use case set; otherwise, the dimension parameters are modified to update the initial template of the use case set until the target use case is obtained when the entity data is input, and the target template of the use case set is output; Generate test cases using the use case set target template to obtain a use case set with multiple attributes; The test set is obtained based on a set of use cases with multiple attributes.

3. The method according to claim 1, characterized in that The interface includes multiple interfaces, and the method of configuring the interface includes: Set the interface name, interface address, interface request parameters, header information, interface checksum field, and field path of each interface.

4. The method according to claim 3, characterized in that Match the test suite to the preconfigured interface, including: Set the task name, corresponding interface, number of concurrent tasks corresponding to the test set, set the filtering method of the test case set in the test set, and set the scheduled task; Among them, each task matches multiple test sets.

5. The method according to claim 4, characterized in that Set the filter method for the test case set in the test set, including at least one of the following: Divide the test set into batches according to the specified batch size, and then obtain a percentage of data from each batch; Query all use cases through SQL, sort by use case address, and filter use cases by index range.

6. The method according to claim 1, characterized in that The test results include: Use case progress, success rate, and results of executed use cases.

7. The method according to claim 1, characterized in that Also includes: Based on different tasks, the use case set is operated; The operating the use case set includes: Edit, delete, view, download, and retry the use case set if failed.

8. A device for executing test cases for a large language model, characterized in that: The device comprises: A generation module, used to generate a test set based on preset rules; the test set includes a plurality of use case sets with corresponding functions, and each use case set includes a plurality of test cases; A matching module, matching the test set with a preconfigured interface according to a test task; The test module is used for the test set to test multiple tasks in the test task through the matching interface to obtain the test results.

9. The device according to claim 8, characterized in that The generating module comprises: A matching unit, used to match a corresponding use case set initial template according to the required use case set type; A determination unit, configured to determine a dimension parameter according to a test scenario, and add the dimension parameter to the initial template of the use case set to obtain a first template of the use case set; An output unit is used to test the first template of the use case set using random entity data, and if a target use case is obtained, the first template of the use case set is used as the target template of the use case set; otherwise, the dimension parameters are modified to update the initial template of the use case set until the target use case is obtained when the entity data is input, and the target template of the use case set is output; A generating unit, used to generate test cases using the use case set target template to obtain a use case set with multiple attributes; The acquisition unit is used to obtain a test set based on a set of use cases with multiple attributes.

10. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

Images