Large model test method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a large model test method and device, computer equipment, a readable storage medium and a program product, and the method comprises the steps: responding to a test instruction, and reading at least one corresponding test task from a task queue; segmenting and / or merging the at least one test task to obtain a processed test task, and obtaining access information of test data corresponding to the processed test task; sending the processed test task and the access information to a computing node of a large model corresponding to the test instruction; and the computing node is used for acquiring test data according to the access information so as to execute the test of the processed test task. By adopting the method, the test efficiency and accuracy of the large model can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a large model testing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] Artificial intelligence big model (AI big model) is a machine learning model with large-scale parameters and complex structure. With the development of big model technology, its application scenarios are becoming more and more extensive, and domain big models in various industries have been developed. Domain big model refers to a large language model that is trained and optimized by using a general big model as the base model and adding domain knowledge of a specific field or industry. Compared with the general language model, the domain big model is more focused on the knowledge and skills of a specific field, and has higher domain professionalism and practicality.

[0003] Large model testing is a key link to ensure model performance, stability and reliability. For general large models, testing generally first determines the test problems (such as functional testing, non-functional testing) based on indicators. However, for domain large models, due to their particularities, general testing methods and indicators are no longer applicable to domain large models. Specifically, general large models have great universality in the design of test cases and often rely on predefined templates and rules. However, due to the lack of vertical in-depth research in a certain field, the testing methods of general large models for domain large models often cannot fully cover all possible professional scenarios, which limits their ability to deal with unknown or professional problems, resulting in reduced accuracy, and domain large models are more sensitive to the accuracy of the results. Summary of the invention

[0004] Based on this, it is necessary to provide a large model testing method, device, computer equipment, computer readable storage medium and computer program product to address the above technical problems.

[0005] In a first aspect, the present application provides a large model testing method. The method comprises:

[0006] In response to the test instruction, reading at least one corresponding test task from the task queue;

[0007] Splitting and / or merging the at least one test task to obtain a processed test task, and acquiring access information of the test data corresponding to the processed test task;

[0008] The post-processing test task and the access information are sent to the computing node of the large model corresponding to the test instruction; the computing node is used to obtain test data according to the access information to execute the test of the post-processing test task.

[0009] In one embodiment, the splitting and / or merging of the at least one test task to obtain a processed test task includes:

[0010] Determining the amount of data required for each of the test tasks, and detecting overlap between the data required for each of the test tasks;

[0011] According to the data volume and the overlap detection result, the at least one test task is split and / or merged to obtain a processed test task.

[0012] In one embodiment, the splitting and / or merging of the at least one test task according to the data volume and the overlap detection result to obtain a processed test task includes:

[0013] When the amount of data required by the test task exceeds a first threshold, splitting the test task;

[0014] When the data volumes of at least two test tasks are smaller than a second threshold value, and / or there is overlap between the data, the at least two test tasks are merged.

[0015] In one embodiment, the method further comprises:

[0016] Determine to split the at least one test task to obtain the processed test task; the processed test task includes a plurality of subtasks;

[0017] The access information of each subtask and the test data corresponding to the subtask is sent to the distributed computing node of the large model respectively; the distributed computing node includes multiple computing nodes, each computing node is used to obtain test data according to the access information corresponding to a subtask to execute the test of the subtask.

[0018] In one embodiment, the obtaining access information of the test data corresponding to the processed test task includes:

[0019] Sending a query request to an object storage; the query request carries a task tag and a task label of the processed test task, and the object storage is used to store test data;

[0020] The shared link of the test data returned by the object storage is received as the access information.

[0021] In one embodiment, the method further comprises:

[0022] By configuring the plug-in, add, delete or modify test tasks in the task queue;

[0023] And, modify the configuration file through the configuration plug-in; the configuration file is configured with the splitting and merging strategy of the test task.

[0024] In a second aspect, the present application also provides a large model testing device. The device comprises:

[0025] A task acquisition module, used for reading at least one corresponding test task from the task queue in response to a test instruction;

[0026] A task processing module, used to split and / or merge the at least one test task to obtain a processed test task, and obtain access information of the test data corresponding to the processed test task;

[0027] A task sending module is used to send the post-processing test task and the access information to the computing node of the large model corresponding to the test instruction; the computing node is used to obtain test data according to the access information to execute the test of the post-processing test task.

[0028] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0029] In response to the test instruction, reading at least one corresponding test task from the task queue;

[0030] Splitting and / or merging the at least one test task to obtain a processed test task, and acquiring access information of the test data corresponding to the processed test task;

[0031] The post-processing test task and the access information are sent to the computing node of the large model corresponding to the test instruction; the computing node is used to obtain test data according to the access information to execute the test of the post-processing test task.

[0032] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0033] In response to the test instruction, reading at least one corresponding test task from the task queue;

[0034] Splitting and / or merging the at least one test task to obtain a processed test task, and acquiring access information of the test data corresponding to the processed test task;

[0035] The post-processing test task and the access information are sent to the computing node of the large model corresponding to the test instruction; the computing node is used to obtain test data according to the access information to execute the test of the post-processing test task.

[0036] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0037] In response to the test instruction, reading at least one corresponding test task from the task queue;

[0038] Splitting and / or merging the at least one test task to obtain a processed test task, and acquiring access information of the test data corresponding to the processed test task;

[0039] The post-processing test task and the access information are sent to the computing node of the large model corresponding to the test instruction; the computing node is used to obtain test data according to the access information to execute the test of the post-processing test task.

[0040] The above-mentioned large model testing method, device, computer equipment, storage medium and computer program product, in response to the test instruction, reads at least one corresponding test task from the task queue; splits and / or merges at least one test task to obtain a processed test task, and obtains access information of the test data corresponding to the processed test task; sends the processed test task and access information to the computing node of the large model corresponding to the test instruction, and the computing node is used to obtain test data according to the access information to perform the test of the processed test task. This method caches test tasks through the task queue, and accurately manages test tasks through the task segmentation and task merging strategies of the task server. On the one hand, it enables the domain large model to be accurately identified in the professional test. On the other hand, it can ensure the rational use of resources and rapid access to data, thereby improving the testing efficiency and accuracy of the large model. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 An application environment diagram of a large model testing method in an embodiment;

[0042] Figure 2 A schematic diagram of a large model testing method in one embodiment;

[0043] Figure 3 A schematic diagram of a large model testing method in another embodiment;

[0044] Figure 4 A schematic diagram of an architecture for implementing a large model testing method in one embodiment;

[0045] Figure 5 is a structural block diagram of a large model testing device in one embodiment;

[0046] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0048] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein.

[0049] The large model testing method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the task server can be implemented with an independent server or a server cluster composed of multiple servers. Among them, the task server communicates with the task queue and the computing node of the large model through the network. Specifically, the task server responds to the test instruction triggered by the user, reads at least one corresponding test task from the task queue; and splits and / or merges at least one test task to obtain a processed test task, and obtains access information of the test data corresponding to the processed test task; sends the processed test task and access information to the computing node of the large model corresponding to the test instruction; the computing node obtains test data according to the access information to perform the test of the processed test task. The test tasks are cached by the task queue, and the task splitting and task merging strategies of the task server are used to accurately manage the test tasks, so that the domain large model can be accurately identified in the professional test.

[0050] In one embodiment, Figure 2 As shown, a large model testing method is provided, which is applied to Figure 1 The task server in the example is used as an example to illustrate the following steps:

[0051] Step S210 , in response to a test instruction, reading at least one corresponding test task from a task queue.

[0052] The test instruction carries at least the task label of the test task and the information of the large model to be tested, and may also carry the task type, the required data volume, etc. The task label indicates the application scenario of the task, such as text translation, fault maintenance, etc. The task type includes general tasks and professional tasks.

[0053] Among them, the task queue stores multiple test tasks, each test task has a corresponding task label, task type, required data volume, task priority, etc. It can also have detailed description information and corresponding test indicators, that is, scoring requirements, such as accuracy, recall rate, etc.

[0054] Specifically, the task queue is used to cache test tasks, and tasks are divided into general tasks and professional tasks. Users can initialize the task queue in the configuration file. The format of each task is:

[0055] Task: [TaskID, TaskType, TaskLabel, Level, Size, Information, Metrics, Result].

[0056] Among them, TaskID identifies the uniqueness of the test task; TaskType indicates the task type, which is divided into general tasks and professional tasks; TaskLabel indicates the task label, such as text translation, fault maintenance, etc., and the big model capabilities are classified according to the specific needs of the test, which is the same as SampleLabel in object storage; Level indicates the task priority, and tasks with high priority are tested first; Size indicates the required data set, that is, the required data set size. For example, if the test of a certain task requires 5,000 data, then Size is 5,000; Metrics indicates the test indicator, that is, the scoring requirement; Information indicates detailed description information, and Result is used to record the returned test results after the big model completes the test. In the task queue, Result is empty. Test tasks that have been read by the task server can be deleted from the task queue.

[0057] In a specific implementation, the task server receives the test instruction from the user, and reads the matching test task from the task queue according to the task information (including at least the task tag) carried in the test instruction.

[0058] In some embodiments, if the test instruction carries information such as the task type and the amount of data required in addition to the task tag, the corresponding task can be read directly from the task queue. In some embodiments, if the test instruction only carries the task tag, the task server can determine the task type, the amount of data required for each task type, etc. based on the task tag, and then read the corresponding task from the task queue.

[0059] Step S220, splitting and / or merging at least one test task to obtain a processed test task, and obtaining access information of the test data corresponding to the processed test task.

[0060] The access information is used to provide a method for obtaining the test data. For example, the access information may be a shared link of the test data.

[0061] It is understandable that a test task may require multiple or even a large number of data samples for testing, some test tasks only need to select a data sample for testing, and some test tasks may overlap. Therefore, this application proposes a task segmentation and merging strategy to accurately manage test tasks, so that the domain large model can be accurately identified in professional testing, while also improving test efficiency.

[0062] Specifically, the amount of data required for each test task can be determined separately, and the overlap between the data required for each test task can be detected. According to the amount of data required for each test task and the overlap detection results, at least one test task is split and / or merged to obtain a processed test task.

[0063] Step S230, sending the processed test task and access information to the computing node of the large model corresponding to the test instruction; the computing node is used to obtain test data according to the access information to execute the test of the processed test task.

[0064] Specifically, the test instruction also carries information about the large model to be tested, which can be used to determine on which computing node to perform the test. After completing the splitting and / or merging of the test task, the access information of the processed test task and the test data corresponding to the processed test task is sent to the computing node of the large model corresponding to the test instruction, so that the computing node obtains the test data corresponding to the processed test task according to the access information, thereby testing the large model through the test data. After completing the test, the computing node can return the test results to the task server. In some embodiments, the computing node can also fill in the test results in the Result of the test task and return them to the task server together.

[0065] In some embodiments, test data can also be shared between computing nodes of a large model by sharing access information, that is, when a computing node needs to transfer a test task to another computing node, data can be shared between computing nodes using access information. The sharing between computing nodes can be data sharing between computing nodes of the same large model or data sharing between computing nodes of different large models.

[0066] In the above-mentioned large model testing method, in response to the test instruction, at least one corresponding test task is read from the task queue; at least one test task is split and / or merged to obtain a processed test task, and access information of the test data corresponding to the processed test task is obtained; the processed test task and the access information are sent to the computing node of the large model corresponding to the test instruction, and the computing node is used to obtain the test data according to the access information to perform the test of the processed test task. This method caches test tasks through the task queue, and accurately manages test tasks through the task segmentation and task merging strategies of the task server. On the one hand, it enables the domain large model to be accurately identified in the professional test. On the other hand, it can ensure the rational use of resources and rapid access to data, thereby improving the testing efficiency and accuracy of the large model.

[0067] In an exemplary embodiment, the above step S220 performs segmentation and / or merging processing on at least one test task to obtain a processed test task, including:

[0068] Step S221, determining the amount of data required for each test task, and detecting the overlap between the data required for each test task;

[0069] Step S222: split and / or merge at least one test task according to the data volume and the overlap detection result to obtain a processed test task.

[0070] In the specific implementation, different test tasks require different amounts of data. The more data there is, the longer it takes to execute the test. If the amount of data is small, the frequent transmission of a single small task to a large model will result in too low throughput and low test efficiency. Therefore, the amount of data can be used as the basis for splitting or merging test tasks. On the other hand, some test tasks will have overlapping data, that is, they reference a large amount of data with the same data label (similar to the task label, representing the application scenario), which will result in a large amount of data being repeatedly tested, causing resource redundancy. Therefore, the data required for each test task can also be checked for overlap, and the overlap detection result can be used as another basis for splitting or merging test tasks. According to at least one condition of the amount of data required for each test task and the overlap detection result between the data required for each test task, at least one test task is split and / or merged to obtain a processed test task.

[0071] In this embodiment, by determining the amount of data required for the test task and the overlap between the data, the splitting / merging strategy for the read test task is determined through two dimensions, thereby improving resource utilization, reducing computing costs, and speeding up the test process.

[0072] In an exemplary embodiment, in the above step S222, at least one test task is split and / or merged according to the data volume and overlap detection results to obtain a processed test task, including: when the data volume required for the test task exceeds a first threshold, the test task is split; when the data volume of at least two test tasks is less than a second threshold, and / or there is overlap between the data, at least two test tasks are merged.

[0073] The first threshold is greater than or equal to the second threshold.

[0074] Specifically, since the reasoning of a large language model takes a long time and the number of data sets to be evaluated is large, the time overhead of running an evaluation task in series is often very large. Therefore, this embodiment sets a first threshold and splits the test task whose data volume is greater than the first threshold to divide the test task into several subtasks, each of which contains n samples (the splitting strategy and subtask size can be customized in the configuration file).

[0075] For some tasks, the data volume may be very small, less than the second threshold, resulting in a single small task being frequently delivered to the large model, which will lead to low throughput efficiency. For some tasks, there will be overlapping data, which will lead to repeated testing of a large amount of data, resulting in resource redundancy. Therefore, this application also proposes a task merging strategy to solve this problem. Specifically, you can customize the merging strategy and the size of the merged tasks in the configuration file, and then merge multiple small tasks and deliver them to the large model for testing at one time. That is: Task:{Subtask1, Subtask2, …}.

[0076] In some embodiments, the data volume of at least two test tasks being smaller than the second threshold value indicates that the data volume of each of the at least two test tasks is smaller than the second threshold value.

[0077] In some embodiments, when the data volume of at least two test tasks is less than a second threshold, and / or there is overlap between the data, after merging the at least two test tasks, the data volume of the processed test task obtained by the merged processing can be further detected. If the data volume of the processed test task is greater than the first threshold, the step of splitting the processed test task is executed, that is, first merging and then splitting to obtain the final processed test task.

[0078] In this embodiment, by splitting the test tasks whose data volume exceeds the first threshold, a large test task is split into multiple computing nodes for running, and computing resources are efficiently used, thereby accelerating the test process. By merging test tasks whose data volume is less than the second threshold and / or have overlapping data, computing resources for model reasoning and evaluation are used to reduce the amount of computing, ensuring efficient testing with limited resources.

[0079] In an exemplary embodiment, the method also includes: determining to split at least one test task to obtain a processed test task; the processed test task includes multiple subtasks; sending access information of each subtask and the test data corresponding to the subtask to the distributed computing nodes of the large model; the distributed computing nodes include multiple computing nodes, each computing node is used to obtain test data according to the access information corresponding to a subtask to execute the test of the subtask.

[0080] Specifically, when the test task is split, it is divided into several subtasks, each of which contains n samples (the splitting strategy and subtask size can be customized in the configuration file). Each subtask and the access information of the test data corresponding to the subtask will be assigned to a computing node of the large model for testing, so that each computing node will obtain the test result of a subtask. Each computing node of the large model returns the test result of the subtask to the task server, and the task server integrates the test results of the same large task to obtain the test result of the test task before splitting.

[0081] In this embodiment, by dividing the test task into multiple subtasks and assigning them to different computing nodes, the tasks can be processed in parallel, thereby significantly shortening the overall test time. This parallel processing method can effectively utilize distributed computing resources and improve the speed and efficiency of the test.

[0082] It can be understood that for the testing of large domain models, as the size of the data set in the professional field increases, the efficiency of the test will gradually decrease, and the consumed resources and labor costs will also increase accordingly. Based on this, the present application also proposes a method that can improve efficiency and reduce costs. Correspondingly, in an exemplary embodiment, the access information of the test data corresponding to the post-processing test task is obtained in the above step S220, including: sending a query request to the object storage; the query request carries the task label and task label of the post-processing test task, and the object storage is used to store the test data; receiving the shared link of the test data returned by the object storage as the access information.

[0083] Among them, Object Storage is a storage architecture that processes, stores and retrieves data in units of objects.

[0084] Specifically, the test data stored in the object storage can be set as a four-tuple data, represented as: [SampleID, SampleLabel, Prompt, Answer]. SampleID represents the data ID, and each piece of data has a unique ID identifier; SampleLabel represents the data label. If a piece of data can be used for multiple different test cases, there will be multiple labels. For example, a piece of data can be used to test fault maintenance capabilities and knowledge assistant capabilities. Prompt and Answer represent a question-answer pair. Prompt represents the variable input into the large model, and Answer can be regarded as supervision information, which is used to compare with the output results of the large model to verify the performance of the large model.

[0085] After completing the splitting and / or merging of the test task, the task server sends a query request to the object storage to obtain the shared link of the test data corresponding to the processed test task. When there are multiple test data, there are multiple corresponding shared links. The shared link is further sent to the computing node of the large model. The computing node requests the test data from the object storage according to the shared link and performs the test. After the test is completed, the test status and test results are returned to the task server. Among them, the data download strategy based on the shared link is: the shared link communication interface adopts the HTTP / HTTPS standard protocol.

[0086] In some embodiments, the query request may also carry a signature. After receiving the query request from the task server, the object storage may also first authenticate the query request, that is, calculate the signature based on the server address host, user AccessKeyId, file path filepath, expiration time Expires and other information in the query request, and compare it with the signature carried by the user. If the comparison is consistent, the authentication is passed, otherwise the request is rejected. Only users who have passed the signature authentication can use the shared link to download files, and when the request time exceeds the expiration time, the shared link becomes invalid and the shared link cannot be used to download files.

[0087] In this embodiment, an object storage module is used to save test data, and tags are used to effectively identify data sets to facilitate management and access. At the same time, data is shared based on shared links, which can reduce the bandwidth and memory usage of data during the interaction process, improve efficiency, and reduce labor costs.

[0088] It can be understood that with the frequent updates of the professional knowledge base of the domain big model and the variety of materials in the knowledge base, the big model is also iterating rapidly, and the test method needs to be able to adapt to changes quickly to ensure test coverage and depth. However, the test method of the general big model often appears to be insufficiently flexible and intelligent when adapting to this rapidly changing environment. Based on this, the present application also proposes a method that can improve the flexibility and intelligence of the test to solve this problem. Correspondingly, in an exemplary embodiment, the method also includes: adding, deleting or modifying test tasks in the task queue by configuring the plug-in; and, modifying the configuration file by configuring the plug-in; the configuration file is configured with the splitting and merging strategies of the test tasks.

[0089] In the specific implementation, the plug-in configuration strategy proposed in this application refers to the interface of the test task. During the test process, the user can modify the configuration file by calling the interface of the configuration plug-in, and dynamically add, delete or modify the test task to the task queue as needed, so as to dynamically implement the customized evaluation tasks and indicators. The interface definition is as follows: plugIn{TaskID, TaskType, TaskLabel,Level, Size, Information,Metrics}. Among them, the definition of each parameter is the same as the definition of each parameter of the test task in the aforementioned embodiment, which will not be repeated here.

[0090] In this embodiment, by adding a plug-in configuration strategy, with the frequent update of the professional knowledge base, it is convenient for users to add, delete and modify tasks online at any time, and flexibly adjust the test indicators of large models, which greatly improves the flexibility and intelligence of the test.

[0091] refer to Figure 3 , is a flow chart of a large model testing method provided by another embodiment. In this embodiment, the following steps are included:

[0092] Step S310, in response to the test instruction, reading at least one corresponding test task from the task queue;

[0093] Step S320, determining the amount of data required for each test task, and detecting the overlap between the data required for each test task;

[0094] Step S330, splitting and / or merging at least one test task according to the data volume and the overlap detection result to obtain a processed test task;

[0095] Specifically, it includes: when the amount of data required for the test task exceeds a first threshold, splitting the test task; when the amount of data of at least two test tasks is less than a second threshold, and / or there is overlap between the data, merging at least two test tasks.

[0096] Step S340, sending a query request to an object storage; the query request carries the task tag and task label of the processed test task, and the object storage is used to store the test data;

[0097] Step S350, receiving a sharing link of the test data returned by the object storage;

[0098] Step S360, sending the processed test task and the shared link to the computing node of the large model corresponding to the test instruction; the computing node is used to obtain test data according to the shared link to perform the test of the processed test task;

[0099] Step S370, when the post-processing test task includes multiple segmented subtasks, receiving the test results of each subtask returned by multiple computing nodes, integrating the test results of each subtask, and obtaining the test result of the post-processing test task.

[0100] In this embodiment, the test tasks are cached by the task queue, and the task splitting and task merging strategies of the task server are used to accurately manage the test tasks. On the one hand, the domain large model can be accurately identified in the professional test, and on the other hand, it can ensure the rational use of resources and rapid access to data, thereby improving the test efficiency and accuracy of the large model. The object storage module is used to save the test data, and the data set is effectively identified by using tags to facilitate management and access. At the same time, data is shared based on shared links, which can reduce the bandwidth and memory occupied by data during the interaction process, improve efficiency, and reduce labor costs.

[0101] In one embodiment, in order to facilitate those skilled in the art to understand the embodiments of the present application, the following will be described with reference to specific examples of the accompanying drawings. Figure 4 , shows a schematic diagram of an architecture for implementing a large model testing method, including the following functional modules: a data set module, a data preprocessing module, an object storage, a task queue, a task server, a computing node, and a plug-in. Among them:

[0102] The dataset module stores data used for testing, which is divided into general datasets and professional datasets.

[0103] The data preprocessing module is used to normalize and clean the test data.

[0104] Object storage is mainly used for: (1) saving preprocessed test data; (2) returning the shared link of the queried data based on the query request; (3) providing data download based on the download request.

[0105] Task queue, used to cache test tasks and divide tasks into general tasks and professional tasks.

[0106] Task server, mainly used for:

[0107] (1) Receive test instructions sent by users;

[0108] (2) Send a query request to the object storage to obtain the shared link of the test data;

[0109] (3) A shared link that sends data to the computing nodes of the large model;

[0110] (4) Receive the test results returned by the computing nodes of the large model;

[0111] (5) Read the test task from the task queue;

[0112] (6) Implement task splitting and task merging strategies.

[0113] The computing nodes of large models are mainly used for:

[0114] (1) Receive test tasks from the task server and return test results;

[0115] (2) Request test data from object storage;

[0116] (3) Execute test tasks;

[0117] (4) Share data links between different large model computing nodes.

[0118] Plugins are used to save plugin configuration files, allowing users to add, delete, and modify tasks online at any time, flexibly adjust test indicators for large models, and adjust task splitting / merging strategies.

[0119] Regarding dataset design: Domain-based models are more adaptable in specific fields, but may be relatively weak in other fields. In actual use, users can be allowed to ask some non-professional questions, so both professional and general datasets are needed, including but not limited to the following data:

[0120] a. Historical test case data: test case data that has been executed in the past, including test case descriptions, expected results, etc.

[0121] b. Requirements documents and design documents: Obtain the project's requirements documents, design documents, and specifications, etc. These documents usually contain requirements for functions, performance, security, etc.

[0122] c. Code base and change log: Collect the code base and change log of the project to understand the structure, function and change history of the code. By analyzing the code, potential test points can be extracted.

[0123] d. User feedback and fault reports: Collect problems and requirements in actual product use.

[0124] e. Industry standards and specifications: Refer to industry standards and specifications, such as software testing standards, security standards, etc., to ensure that the test covers important aspects and potential risk points.

[0125] Object storage design: Stores preprocessed test data. Each sample in the test data is a four-tuple data: Dataset: [SampleID, SampleLabel, Prompt, Answer]. SampleID represents the data ID. Each piece of data has a unique ID. SampleLabel represents the data label. A piece of data can be used for multiple different test cases, so there are multiple labels. For example, a piece of data can be used to test fault maintenance capabilities or knowledge assistant capabilities. Prompt and Answer represent a question-answer pair.

[0126] Task queue design: The task queue is used to cache test tasks and divide tasks into general tasks and professional tasks. Users can initialize the task queue in the configuration file.

[0127] Plug-in design: The plug-in configuration strategy of this application is to interface the test tasks. During the test process, users can modify the configuration files by calling the interface, and dynamically add, delete or modify test tasks to the task queue as needed to dynamically implement custom evaluation tasks and indicators.

[0128] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0129] Based on the same inventive concept, the embodiment of the present application also provides a large model testing device for implementing the large model testing method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more large model testing device embodiments provided below can refer to the limitations of the large model testing method above, and will not be repeated here.

[0130] In one embodiment, Figure 5 As shown, a large model testing device is provided, including: a task acquisition module 510, a task processing module 520 and a task sending module 530, wherein:

[0131] The task acquisition module 510 is used to read at least one corresponding test task from the task queue in response to the test instruction;

[0132] The task processing module 520 is used to split and / or merge at least one test task to obtain a processed test task, and obtain access information of the test data corresponding to the processed test task;

[0133] The task sending module 530 is used to send the post-processing test task and access information to the computing node of the large model corresponding to the test instruction; the computing node is used to obtain test data according to the access information to execute the test of the post-processing test task.

[0134] In one embodiment, the task processing module 520 is also used to determine the amount of data required for each test task, and to detect the overlap between the data required for each test task; based on the data amount and overlap detection results, at least one test task is split and / or merged to obtain a processed test task.

[0135] In one embodiment, the task processing module 520 is also used to split the test task when the amount of data required for the test task exceeds a first threshold; and to merge at least two test tasks when the amount of data of at least two test tasks is less than a second threshold and / or there is overlap between the data.

[0136] In one embodiment, the task processing module 520 is further used to determine to split at least one test task to obtain a processed test task; the processed test task includes a plurality of subtasks;

[0137] The task sending module 530 is also used to send access information of each subtask and the test data corresponding to the subtask to the distributed computing node of the large model respectively; the distributed computing node includes multiple computing nodes, each computing node is used to obtain test data according to the access information corresponding to a subtask to execute the test of the subtask.

[0138] In one embodiment, the task processing module 520 is also used to send a query request to the object storage; the query request carries the task tag and task tag of the processed test task, and the object storage is used to store test data; and the shared link of the test data returned by the object storage is received as access information.

[0139] In one of the embodiments, the device also includes an adjustment module for adding, deleting or modifying test tasks in the task queue by configuring a plug-in; and modifying a configuration file by configuring a plug-in; the configuration file is configured with a splitting and merging strategy for the test tasks.

[0140] Each module in the above-mentioned large model test device can be implemented in whole or in part by software, hardware and their combination. Each module can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0141] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data during the large model test process. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a large model test method is implemented.

[0142] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0143] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0144] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0145] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0146] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0147] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0148] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0149] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A large model testing method, characterized in that: The method comprises: In response to the test instruction, reading at least one corresponding test task from the task queue; Splitting and / or merging the at least one test task to obtain a processed test task, and acquiring access information of the test data corresponding to the processed test task; The post-processing test task and the access information are sent to the computing node of the large model corresponding to the test instruction; the computing node is used to obtain test data according to the access information to execute the test of the post-processing test task.

2. The method according to claim 1, characterized in that: The splitting and / or merging of the at least one test task to obtain a processed test task includes: Determining the amount of data required for each of the test tasks, and detecting overlap between the data required for each of the test tasks; According to the data volume and the overlap detection result, the at least one test task is split and / or merged to obtain a processed test task.

3. The method according to claim 2, characterized in that The step of dividing and / or merging the at least one test task according to the data volume and the overlap detection result to obtain a processed test task includes: When the amount of data required by the test task exceeds a first threshold, splitting the test task; When the data volumes of at least two test tasks are smaller than a second threshold value, and / or there is overlap between the data, the at least two test tasks are merged.

4. The method according to claim 1, characterized in that: The method further comprises: Determine to split the at least one test task to obtain the processed test task; the processed test task includes a plurality of subtasks; The access information of each subtask and the test data corresponding to the subtask is sent to the distributed computing node of the large model respectively; the distributed computing node includes multiple computing nodes, each computing node is used to obtain test data according to the access information corresponding to a subtask to execute the test of the subtask.

5. The method according to claim 1, characterized in that The obtaining access information of the test data corresponding to the processed test task includes: Sending a query request to an object storage; the query request carries a task tag and a task label of the processed test task, and the object storage is used to store test data; The shared link of the test data returned by the object storage is received as the access information.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: By configuring the plug-in, add, delete or modify test tasks in the task queue; And, modify the configuration file through the configuration plug-in; the configuration file is configured with the splitting and merging strategy of the test task.

7. A large model testing device, characterized in that: The device comprises: A task acquisition module, used for reading at least one corresponding test task from the task queue in response to a test instruction; A task processing module, used to split and / or merge the at least one test task to obtain a processed test task, and obtain access information of the test data corresponding to the processed test task; A task sending module is used to send the post-processing test task and the access information to the computing node of the large model corresponding to the test instruction; the computing node is used to obtain test data according to the access information to execute the test of the post-processing test task.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the large model testing method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the large model testing method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the large model testing method according to any one of claims 1 to 6 are implemented.

Images