Large model testing methods and apparatus, electronic devices and storage media
By dividing large model test cases into multiple batches, conducting parallel tests, and requesting computing resources according to resource consumption, the problems of long testing time and low resource utilization of large models are solved, achieving efficient GPU resource utilization and test concurrency.
Patent Information
- Application Number
- CN202611387820.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-08
- Publication Date
- 2026-11-17
AI Technical Summary
Large model testing is time-consuming and has low computing resource utilization. Existing technologies use a static binding mode on GPU servers, which leads to hardware determinism and sacrifices global flexibility, resulting in low overall GPU computing power utilization of the cluster.
By dividing the target test cases into multiple target test batches, parallel testing is carried out. Each batch requests computing resources according to the upper limit of resource consumption, and dynamic scheduling is performed using GPU resource monitoring to achieve parallel testing and efficient resource utilization.
Significantly shorten test duration, improve computing resource utilization, optimize the overall resource scheduling efficiency of GPU clusters, reduce the monopolization of resources by long-tail tasks, and improve test concurrency and stability.
Smart Images

Figure CN123114973A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of testing technology, and in particular to a large-scale model testing method and apparatus, electronic equipment, and storage medium. Background Technology
[0002] With the rapid development of Large Language Model (LLM) technology, intelligent agent applications built on large models are becoming increasingly complex, and their testing and continuous integration (CI) processes face numerous challenges. In current industry practice, there are already some mature technical approaches for the deployment and testing of large model inference services.
[0003] In terms of optimizing the deployment of model inference services, relevant technologies typically provide automated deployment capabilities for specific inference frameworks (such as TensorRT-LLM). The general process of such solutions includes: receiving the user's selection of the target model through an interactive interface; automatically evaluating and selecting appropriate acceleration configuration parameters (such as quantization accuracy, operator fusion strategy, batch size, etc.); generating a corresponding optimized inference engine based on the selected inference framework; subsequently deploying the generated engine in an isolated pre-test environment and allocating corresponding GPU computing resources; adjusting and optimizing the configuration and large model testing based on the pre-test results; and finally releasing the optimized model engine to the online inference environment, while continuously monitoring the performance metrics and operational status of the online service. The main focus of this type of solution is to build an end-to-end automated process from model selection to online deployment, aiming to improve inference performance and deployment efficiency.
[0004] In related technologies, when conducting software testing on large models, especially for codebases that iterate frequently within these models, all code in the large model must be tested after each update. This results in long test execution times and consumes significant computing resources. Furthermore, although modern GPU servers are typically equipped with multiple GPU cards (e.g., eight or more), existing test execution schemes often employ a static binding model: "one process fixed to one card" or "several processes fixed to several cards." This model sacrifices global flexibility to achieve hardware determinism, leading to low overall GPU computing power (i.e., computing resources) utilization across the cluster, and significantly longer end-to-end latency from CI task triggering to the completion of all test cases.
[0005] In summary, large-scale model testing in related technologies suffers from technical problems such as long testing time and low utilization of computing resources. Summary of the Invention
[0006] One of the objectives of this invention is to provide a large model testing method and apparatus, electronic device and storage medium to solve the technical problems of long testing time and low utilization of computing resources in large model testing in related technologies.
[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows: According to one aspect of the embodiments of this application, a large model testing method is provided, comprising: Obtain the target model to be tested; Based on the modification information of the target model, determine all target test cases for testing the target model; Based on the resource consumption of each target test case in all target test cases, all target test cases are divided into multiple target test batches. In the case where multiple target test cases are contained in the same target test batch, the difference between the resource consumption of each target test case in the multiple target test cases is less than a preset difference upper limit. According to the upper limit of resource consumption for each target test batch, the computing resources corresponding to each target test batch are applied for, wherein the upper limit of resource consumption for each target test batch is the amount of resources required by the target test case that consumes the most resources in each target test batch; Each target test batch is tested in parallel using the computing resources corresponding to each target test batch.
[0008] Optionally, as described in the aforementioned large model testing method, determining all target test cases for testing the target model based on the modification information of the target model includes: The modification information is obtained by parsing the specified module file that has been modified in the target model, wherein the modification information includes: file information of the specified module file that has been modified; Based on the modified information and the pre-determined mapping relationship between module files and test cases, all target test cases corresponding to all the specified module files are determined.
[0009] Optionally, as described in the aforementioned large model testing method, determining all target test cases for testing the target model based on the modification information of the target model includes: The modification information is obtained by analyzing whether the third-party dependencies of the target model have changed relative to the third-party dependencies of the previous version of the target model. The modification information includes: modification information used to indicate whether the third-party dependencies of the target model have changed. If, based on the modified information, it is determined that the third-party dependencies of the target model have changed relative to the third-party dependencies of the previous version of the target model, then the test cases corresponding to all module files in the target model are determined as all target test cases.
[0010] Optionally, as described in the aforementioned large-scale model testing method, the step of dividing all target test cases into multiple target test batches according to the resource consumption of each target test case in all target test cases includes: By obtaining the historical test time and GPU requirements for each target test case, the resource consumption of each target test case is determined. According to the GPU requirement of each target test case and the first batch division requirements, all target test cases are divided to obtain multiple target test batches. The first batch division requirements include: when the same target test batch contains multiple target test cases, the difference between the GPU requirements of each target test case in the multiple target test cases is less than a preset difference upper limit.
[0011] Optionally, as described in the aforementioned large-model testing method, the step of dividing all target test cases according to the GPU requirement of each target test case and the first batch division requirements to obtain the multiple target test batches includes: All target test cases are divided according to the GPU requirement for each target test case and the first batch division requirements to obtain multiple specified test batches; If, among the multiple specified test batches, there is a first test batch that does not meet the second batch division requirements, the first test batch is split to obtain multiple second test batches that meet the second batch division requirements. The second batch division requirements include: the number of test cases contained in a test batch is less than or equal to a preset threshold. The plurality of target test batches are obtained based on the second test batch and the remaining specified test batches other than the first test batch among the plurality of specified test batches.
[0012] Optionally, as described in the aforementioned large model testing method, the step of requesting computing resources corresponding to each target test batch according to the upper limit of resource consumption for each target test batch includes: Based on the resource consumption limit of each target test batch, configuration information corresponding to all target test batches is generated. The configuration information is used to define: the Pod specification required to execute the task of each target test batch. The Pod specification required for the task of each target test batch is greater than or equal to the resource consumption limit of each target test batch. The Pod specification includes: number of GPUs, GPU type, CPU resources, and memory resources. According to the Pod specifications of each target test batch, apply for the Pod resources corresponding to each target test batch as the computing resources corresponding to each target test batch.
[0013] Optionally, as described in the aforementioned large model testing method, the step of performing parallel testing on each target test batch using the computing resources corresponding to each target test batch includes: If the Pod resources corresponding to the target test batch are greater than the upper limit of the resource usage of the target test batch, multiple test processes are started within the Pod corresponding to the target test batch, wherein each test process executes one test case. The system acquires real-time monitoring data of the Pod resources, including the memory usage and computing power utilization of each GPU in the Pod resources. If, based on the real-time monitoring data, it is determined that there is an idle GPU in the Pod resources, at least one of the multiple test processes calls the idle GPU to test the first test case to be executed in the target test batch. The idle GPU is defined as one whose video memory usage and computing power utilization are lower than the preset load requirement, and the first test case is a target test case whose GPU can be shared with other target test cases.
[0014] Optionally, as described in the aforementioned large model testing method, the step of performing parallel testing on each target test batch using the computing resources corresponding to each target test batch includes: If the target test case in the target test batch is the second test case, the second test case is tested by using the GPU to which the second test case belongs in the computing resources exclusively. If the target test case in the target test batch is the first test case, the first test case is tested by sharing the GPU to which the first test case belongs in the computing resources.
[0015] Optionally, as described in the large model testing method above, the method further includes: If an anomaly is detected during the testing of the target test case, the computing resources currently occupied by the target test case are cleared according to the preset retry strategy, the target test case is retested, and the test count of the target test case is incremented by 1. If the number of times the target test case is tested exceeds the maximum number of retries, the target test case is marked as a test failure.
[0016] According to another aspect of the embodiments of this application, a large model testing apparatus is also provided, comprising: The acquisition module is used to acquire the target model to be tested; The determination module is used to determine all target test cases for testing the target model based on the modification information of the target model. The partitioning module is used to divide all target test cases into multiple target test batches according to the resource consumption of each target test case in all target test cases. In the case that there are multiple target test cases in the same target test batch, the difference between the resource consumption of each target test case in the multiple target test cases is less than a preset difference upper limit. The application module is used to apply for computing resources corresponding to each target test batch according to the upper limit of resource consumption for each target test batch, wherein the upper limit of resource consumption for each target test batch is the amount of resources required by the target test case that consumes the most resources in each target test batch; The testing module is used to perform parallel testing on each target test batch using the computing resources corresponding to each target test batch.
[0017] According to another aspect of the embodiments of this application, an electronic device is also provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; wherein the memory is used to store a computer program; and the processor is used to execute the method steps of any of the above embodiments by running the computer program stored in the memory.
[0018] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored therein, wherein the computer program is configured to execute the method steps of any of the above embodiments when running.
[0019] This application obtains a target model to be tested; based on the modification information of the target model, determines all target test cases for testing the target model; divides all target test cases into multiple target test batches according to the resource consumption of each target test case, wherein, when multiple target test cases are contained in the same target test batch, the difference between the resource consumption of each target test case in the multiple target test cases is less than a preset difference upper limit; and applies for computing resources corresponding to each target test batch according to the upper limit of the resource consumption of each target test batch, wherein the upper limit of the resource consumption of each target test batch is the amount of resources required by the target test case with the highest resource consumption in each target test batch; through the... The method described above utilizes the computing resources corresponding to each target test batch to perform parallel testing on each target test batch. By dividing all target test cases into multiple target test batches according to resource consumption and applying corresponding computing resources for each batch, parallel testing can be performed, thereby effectively improving test concurrency and reducing test duration. Furthermore, by applying for computing resources corresponding to each target test batch according to its resource consumption limit, each target test batch only uses computing resources greater than or equal to its resource consumption limit. Therefore, each target test case within a target test batch will only be tested using the computing resources corresponding to its respective target test batch, preventing all target test cases from simultaneously and for extended periods consuming large amounts of computing resources, thus improving computing power utilization. Therefore, the method of this application effectively overcomes the technical problems of long testing times and low computing resource utilization in large model testing in related technologies. Attached Figure Description
[0020] Figure 1 This is a flowchart illustrating an optional large-model testing method according to an embodiment of this application; Figure 2 This is a flowchart illustrating another optional large model testing method according to an embodiment of this application; Figure 3 This is a structural block diagram of an optional large model testing device according to an embodiment of this application; Figure 4 This is a structural block diagram of an optional electronic device according to an embodiment of this application. Detailed Implementation
[0021] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.
[0022] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0023] According to one aspect of the embodiments of this application, a large-scale model testing method is provided. Optionally, in this embodiment, the above-described large-scale model testing method can be applied to a hardware environment consisting of a terminal and a server. The server is connected to the terminal via a network and can be used to provide services (such as advertising push services, application services, etc.) to the terminal or clients installed on the terminal. A database can be set up on the server or independently of the server to provide data storage services to the server.
[0024] The aforementioned network may include, but is not limited to, at least one of the following: wired network, wireless network. The aforementioned wired network may include, but is not limited to, at least one of the following: wide area network, metropolitan area network, local area network. The aforementioned wireless network may include, but is not limited to, at least one of the following: Wi-Fi (Wireless Fidelity), Bluetooth. The terminal may not be limited to PC, mobile phone, tablet computer, etc.
[0025] The large model testing method of this application embodiment can be executed by a server, a terminal, or both. Alternatively, the large model testing method of this application embodiment can be executed by a client installed on the terminal.
[0026] Taking the large model testing method in this embodiment, executed by the server, as an example, Figure 1 A large model testing method provided in this application includes the following steps: Step S102: Obtain the target model to be tested.
[0027] The large model testing method in this embodiment can be applied to scenarios requiring efficient execution of large-scale model testing tasks (such as regression testing, performance stress testing, model inference verification, training robustness testing, etc.) on GPU (Graphics Processing Unit) computing clusters. The large model in this embodiment can be a large language model (LLM), a large visual model, or a multimodal large model, etc. Where there is no contradiction, the above-described large model testing method is equally applicable.
[0028] Specifically, users can upload the target model to be tested to the large model testing system that implements the method of this embodiment through a preset interface, so that the large model testing system can obtain the target model.
[0029] Step S104: Based on the modification information of the target model, determine all target test cases for testing the target model.
[0030] In other words, after acquiring the target model, the system immediately parses the list of specific modules involved in the current modification of the target model. Subsequently, the system can consult a pre-built relationship between modules and test cases to identify all test cases directly associated with the modified modules, thus obtaining all target test cases. This mechanism ensures that the test scope is neither excessively redundant nor insufficiently redundant, while accurately covering all potential risk points.
[0031] As an optional implementation, the large model testing method described above can be implemented by determining all target test cases for testing the target model based on the modification information of the target model through the following steps: By parsing the specified module files that have changed in the target model, modification information is obtained. The modification information includes: file information of the specified module files that have changed. Based on the modification information and the pre-determined mapping relationship between module files and test cases, all target test cases corresponding to all specified module files are determined.
[0032] Specifically, upon receiving a code commit or merge request, the system first parses the set of file changes for that commit to identify the specified module files in the target model that have been changed, thereby obtaining the modification information. This modification information includes at least the file information of the specified module files that have been changed. Subsequently, based on a pre-built mapping relationship of "module file → Test Case (i.e., test case, also known as Case) file / ID", all the specified module files involved in the modification information are mapped one by one to the corresponding Test Cases. Finally, all target test cases corresponding to all specified module files are determined, forming an incremental test case set containing all target test cases.
[0033] By analyzing whether the third-party dependencies of the target model have changed relative to the third-party dependencies of the previous version of the target model, modification information is obtained. The modification information includes: modification information used to indicate whether the third-party dependencies of the target model have changed. If, based on the modification information, it is determined that the third-party dependencies of the target model have changed relative to the third-party dependencies of the previous version of the target model, the test cases corresponding to all module files in the target model are determined as all target test cases.
[0034] Specifically, while parsing file changes, it is also necessary to check whether the third-party dependencies of the target model have changed compared to the third-party dependencies of its previous version. This change information is also classified as modification information. When it is determined based on this modification information that the third-party dependencies of the target model have indeed changed, for compatibility reasons, the incremental selection logic based on module file mapping is no longer executed. Instead, all test cases corresponding to all module files in the target model are identified as all target test cases. That is, the incremental test case set is automatically expanded into a full test set to ensure that all related functions are fully verified after the third-party dependency is upgraded or replaced.
[0035] Furthermore, before the target test case parsing step, a test metadata collection and maintenance step is first performed to establish and continuously maintain structured metadata information for each test case (i.e., Case). Each piece of metadata includes at least: the module path corresponding to the Case (i.e., which specific module or directory in the codebase this test case is written for), the type of Agent Server it depends on (i.e., large model applications usually call external Agent services (such as inference servers, vector database services)), the number of GPUs required for execution, the concurrency attribute (shared / exclusive) (i.e., marking whether this Case can run on the same GPU as other Cases simultaneously), historical execution time statistics (i.e., recording how long it took for this Case to be executed several times in the past, such as including but not limited to: average, P90 percentile, P99 percentile, etc.), historical memory usage data (i.e., recording how many GB of memory were used during the peak execution of this Case in history, including average usage, P99 peak, etc.), and the test type (CPU test / GPU test / Server load test, etc.). This metadata provides the data foundation for all subsequent scheduling decisions.
[0036] This embodiment maintains rich metadata for each test case, including module path, GPU requirements, concurrency attributes, historical execution time, and memory data. This ensures that subsequent batch partitioning, resource allocation, and GPU scheduling are all based on data, laying the foundation for fine-grained scheduling. Secondly, based on the mapping relationship between module files and test cases, it allows execution only on test cases affected by code changes, significantly reducing unnecessary duplicate testing and shortening the total CI testing time while ensuring test coverage. Thirdly, through automatic detection of third-party dependency changes and a full-scale testing fallback strategy, it effectively avoids compatibility risks caused by changes in dependency library versions. This maintains the efficiency of incremental testing while ensuring system stability and reliability through full-scale testing in critical scenarios.
[0037] Step S106: According to the resource consumption of each target test case in all target test cases, divide all target test cases into multiple target test batches. Where there are multiple target test cases in the same target test batch, the difference between the resource consumption of each target test case in the multiple target test cases is less than the preset difference upper limit.
[0038] Specifically, after identifying all target test cases, the system proceeds to the test batch partitioning step. The core task of this step is to rationally divide all target test cases into multiple target test batches based on the resource consumption of each target test case, laying the foundation for subsequent CI Pipeline task generation and resource allocation. In this step, each target test case is first identified, and then each target test case is partitioned into batches according to pre-defined resource consumption ranges, resulting in multiple target test batches, each with a corresponding resource consumption range. The resource consumption range can be the number of GPUs required; for example, a range of 1 GPU, 2 GPUs, 4 GPUs, etc. Optionally, a greedy strategy can be used to divide all target test cases into multiple target test batches. Besides the greedy strategy, dynamic programming, integer programming, or machine learning-based predictive models can also be used for batch partitioning to further improve scheduling quality.
[0039] As an optional implementation, the large-scale testing method described above can be implemented by dividing all target test cases into multiple target test batches according to the resource consumption of each target test case in all target test cases, through the following steps: By obtaining the historical test time and GPU requirement of each target test case, the resource consumption of each target test case is determined. According to the GPU requirement of each target test case and the first batch division requirements, all target test cases are divided into multiple target test batches. The first batch division requirements include: when there are multiple target test cases in the same target test batch, the difference between the GPU requirements of each target test case in the multiple target test cases is less than the preset difference limit.
[0040] Specifically, the first batch division requirements stipulate that when the same target test batch contains multiple target test cases, the difference between the number of GPUs required for each target test case must be less than the preset difference limit. This ensures that the target test cases within the same batch (i.e., the target test batch) are highly similar in the number of GPUs, so that the K8S Tekton Task / Pod (container group) corresponding to each batch can accurately apply for GPU resources on demand, avoiding resource fragmentation.
[0041] Furthermore, the following steps can be used to divide all target test cases into multiple target test batches based on the GPU requirements for each target test case and the batch division requirements: Based on the GPU requirements for each target test case and the first batch partitioning requirements, all target test cases are partitioned into multiple designated test batches. If a first test batch does not meet the second batch partitioning requirements among the multiple designated test batches, the first test batch is split to obtain multiple second test batches that meet the second batch partitioning requirements. The second batch partitioning requirements include: the number of test cases in a test batch is less than or equal to a preset threshold. Based on the second test batches and the remaining designated test batches other than the first test batch, multiple target test batches are obtained.
[0042] Specifically, after dividing the system into multiple designated test batches based on GPU requirements, the system further checks whether each designated test batch meets the second batch division requirement, i.e., whether the number of test cases in a test batch is less than or equal to a preset threshold. When there is a first test batch that does not meet this requirement (i.e., the number of cases in a single batch is too large), the system splits the first test batch into multiple second test batches, ensuring that each of the split batches meets the constraint that the number of cases does not exceed the preset threshold. This splitting mechanism essentially further divides the first test batch with a certain type of GPU requirement into multiple micro-batches (i.e., second test batches) to avoid excessively long execution times for a single batch, which could lead to prolonged occupation of scarce GPU resources and cause cluster resource strain. Finally, based on the split second test batches and the remaining designated test batches that originally met the second batch division requirement, all target test batches are merged.
[0043] Furthermore, when the same first test batch is divided into multiple second test batches, the same GPU resources can be used for testing, or different GPU resources can be used for testing. For example, when the first test batch A is divided into multiple second test batches (a1, a2), when the system finds that there are other idle GPUs available for testing that meet the requirements of a2 during the testing of a1, it can use the idle GPU to test a2, instead of having to test a1 and a2 in sequence.
[0044] In this embodiment, precise grouping of GPU demand ensures highly consistent resource requirements for each target test batch, enabling accurate adaptation of Pod resource configurations and significantly reducing resource waste. By introducing preset threshold constraints and further splitting designated test batches with excessive target test cases, it effectively prevents individual long-tail batches from monopolizing GPU resources for extended periods, reducing peak resource usage and improving overall cluster scheduling efficiency. By comprehensively considering historical latency data, it avoids slowing down the overall pipeline by mixing long and short tasks, significantly shortening the total CI test latency and making the testing process run more smoothly and efficiently in a shared K8S cluster.
[0045] Step S108: Apply for computing resources corresponding to each target test batch according to the upper limit of resource consumption for each target test batch. The upper limit of resource consumption for each target test batch is the amount of resources required by the target test case that consumes the most resources in each target test batch.
[0046] After completing the test batch segmentation, the system proceeds to the CI Pipeline task generation and resource allocation steps. For each target test batch, the system does not allocate computing resources based on the sum of the resource requirements of all test cases within that batch. Instead, it allocates resources precisely based on the maximum resource usage limit for that batch. Specifically, the maximum resource usage limit for each target test batch is defined as the amount of resources required by the target test case that consumes the most resources within that batch. In other words, the system uses the target test case that requires the most resources within a batch as the standard to allocate computing resources corresponding to each target test batch. For example, it requests Pod resources of the corresponding specifications (including the number of GPUs, GPU type, CPU / memory, etc.) from the Kubernetes cluster. Because the batch segmentation phase has already ensured that the resource requirements of each target test case within the same batch are highly similar through the first batch segmentation requirements, there is no significant waste between the maximum resource usage limit and the actual demand. Furthermore, the system may not allocate resources for all test batches at once, but rather allocate them as needed during the CI pipeline execution, releasing the corresponding computing resources after each batch of tests is completed. This greatly reduces peak cluster resource usage, making the entire testing process smoother and more economical.
[0047] Step S110: Perform parallel testing on each target test batch using the computing resources corresponding to each target test batch.
[0048] Specifically, after completing the application for computing resources corresponding to each target test batch, the system uses the allocated computing resources to perform parallel testing within different target test batches. In other words, each target test batch can be tested in parallel using its own computing resources. Furthermore, for different target test cases within the same target test batch, it can be determined whether parallel testing is possible based on the usage of computing resources in the target test batch and the amount of resources required for each target test case.
[0049] This application obtains the target model to be tested; based on the modification information of the target model, determines all target test cases for testing the target model; divides all target test cases into multiple target test batches according to the resource consumption of each target test case, wherein, when multiple target test cases are contained in the same target test batch, the difference in resource consumption between the target test cases in the multiple target test cases is less than a preset difference upper limit; according to the upper limit of resource consumption for each target test batch, requests the computing resources corresponding to each target test batch, wherein the upper limit of resource consumption for each target test batch is the amount of resources required by the target test case with the highest resource consumption in each target test batch; and then, through each target test batch... The method of using corresponding computing resources to perform parallel testing for each target test batch effectively improves test concurrency and reduces test duration by dividing all target test cases into multiple target test batches according to resource consumption and requesting corresponding computing resources for each target test batch. Furthermore, by requesting computing resources corresponding to the resource consumption limit of each target test batch, each target test batch only uses computing resources greater than or equal to its resource consumption limit. Therefore, each target test case within a target test batch will only be tested using the computing resources corresponding to its respective target test batch, preventing all target test cases from simultaneously and for extended periods consuming large amounts of computing resources, thus improving computing power utilization. Therefore, the method of this application effectively overcomes the technical problems of long testing time and low computing resource utilization in large model testing in related technologies.
[0050] As an optional implementation, the large-model testing method described above can be implemented by applying for computing resources corresponding to each target test batch according to the resource consumption limit of each target test batch through the following steps: Based on the resource consumption limit of each target test batch, configuration information corresponding to all target test batches is generated. The configuration information is used to define: the Pod specification required to execute the task of each target test batch. The Pod specification required for the task of each target test batch is greater than or equal to the resource consumption limit of each target test batch. The Pod specification includes: number of GPUs, GPU type, CPU resources and memory resources. Specifically, after completing the test batch segmentation, the system proceeds to the CI Pipeline (Continuous Integration Pipeline) task generation and resource allocation steps. First, based on the segmentation results of all target test batches, the CIPipeline configuration is automatically generated or updated. During configuration generation, for each target test batch's corresponding Tekton Task (i.e., a task unit, used to define a specific operation step that runs automatically in a container), the system defines the required Pod specifications based on the resource consumption limit of that batch. Specifically, the Pod specifications required for each target test batch's tasks must be greater than or equal to the resource consumption limit of that batch. That is, the system uses the resource requirements of the target test case with the largest resource requirement within a target test batch as a benchmark, and may incorporate a certain safety margin to ultimately determine the Pod specifications for that batch. The complete definition of the Pod specifications includes: the number of GPUs, GPU type, CPU resources, and memory resources, ensuring that the allocated Pods can fully meet the execution requirements of all test cases within that batch.
[0051] According to the Pod specifications of each target test batch, request the Pod resources corresponding to each target test batch as the computing resources for each target test batch.
[0052] Specifically, during the actual execution of the CI pipeline, the system requests corresponding Pod resources from the K8S cluster as needed, according to the Pod specifications defined for each target test batch, as computing resources for each target test batch.
[0053] This embodiment precisely defines Pod specifications based on the maximum resource usage limit for each target test batch, ensuring a high degree of match between resource configuration and test requirements, thus avoiding resource waste.
[0054] like Figure 2 As shown, as an optional embodiment, the large model testing method described above performs parallel testing on each target test batch using the computing resources corresponding to each target test batch, including: Step S202: If the Pod resources corresponding to the target test batch are greater than the upper limit of the resource usage of the target test batch, multiple test processes are started within the Pod corresponding to the target test batch, and each test process executes one test case.
[0055] Specifically, after allocating Pod resources for each target test batch, the system first starts a multi-process / multi-worker test execution environment within the Pod for each target test batch, based on Pytest and its extended plugins (including xdist, xprocess, monkeypatch, etc.). If the Pod resources exceed the resource limit for that target test batch, the system will start multiple test processes within that Pod. Each test process is responsible for executing one test case, thereby achieving high concurrency of multiple target test cases within a single target test batch and fully utilizing the allocated GPU computing power.
[0056] Step S204: Obtain real-time monitoring data of Pod resources, including: memory usage and computing power utilization of each GPU in the Pod resources.
[0057] Specifically, during the test run, the system continuously acquires real-time monitoring data of each GPU card in the Pod resources through GPU monitoring tools or custom monitoring modules, including the memory usage and computing power utilization of each GPU card.
[0058] Step S206: If it is determined from real-time monitoring data that there is an idle GPU in the Pod resources, at least one of the multiple test processes calls the idle GPU to test the first test case to be executed in the target test batch. Here, the idle GPU is one whose memory usage and computing power utilization are lower than the preset load requirement, and the first test case is a target test case whose GPU can be shared with other target test cases.
[0059] Specifically, based on this real-time monitoring data, the system determines whether each GPU in the Pod resources is idle, i.e., whether the memory usage and computing power utilization are lower than the preset load requirements. When an idle GPU is determined to exist, the system combines the prior GPU usage data (i.e., historical memory usage data) in the target test case metadata with the real-time monitoring data to dynamically allocate the first test case to an idle GPU with low current load and sufficient memory for testing. Furthermore, the first test case specifically refers to target test cases whose concurrency attribute is marked as shared, meaning that the GPU to which it belongs can be shared with other target test cases. The system only performs dynamic migration scheduling for this type of case; for test cases marked as exclusive, the exclusive constraint is strictly followed, and they do not participate in this type of dynamic allocation.
[0060] This embodiment utilizes a multi-process parallel execution mechanism to fully leverage the multi-process capabilities within a Pod, significantly improving the concurrent execution efficiency of test cases within the target test batch and shortening the overall test time. Through continuous real-time GPU monitoring and dynamic scheduling, test cases are preferentially allocated to idle GPUs with low load and sufficient video memory, achieving load balancing among multiple GPUs and effectively avoiding resource imbalance caused by some GPUs being idle for extended periods while others are partially overloaded. Combining prior GPU usage data with allocation decisions further reduces the risk of insufficient video memory and improves the success rate and stability of test execution.
[0061] As an optional implementation, the large model testing method described above involves performing parallel testing on each target test batch using the computing resources corresponding to each target test batch, including: When the target test case in the target test batch is the second test case, the second test case is run exclusively on the GPU to which it belongs. In other words, for the target test case marked as exclusive in the target test batch, i.e., the second test case, the system uses an exclusive running mode. Specifically, after the second test case is assigned to its GPU, that GPU is not allowed to run in parallel with any other test case during the execution of the second test case; that is, the GPU is completely exclusively used by the second test case. This strategy avoids mutual interference caused by memory contention or insufficient computing resources due to multiple cases running in parallel, thus ensuring the accuracy and repeatability of the test results, especially for test scenarios with high memory usage or high requirements for computing power stability.
[0062] When the target test case in the target test batch is the first test case, the first test case is tested using a shared execution method on the GPU to which it belongs. In other words, for the target test case marked as "shared" in the target test batch, i.e., the first test case, the system uses a shared execution method. Specifically, the first test case can share the GPU's memory and computing power resources with other first test cases on the same GPU, but this sharing is not unlimited. The system can strictly constrain the number of first test cases running simultaneously on the GPU based on the preset maximum number of parallel cases per card threshold in the global configuration file, ensuring that the number of parallel executions does not exceed this limit. This ensures full utilization of GPU resource reserves while avoiding memory overflow or severe performance fluctuations caused by excessive parallelism, balancing resource utilization and execution stability.
[0063] In this embodiment, differentiated concurrency control using exclusive and shared methods is employed for the first and second test cases, respectively. This ensures the reliability and accuracy of resource-sensitive tests while maximizing resource reuse efficiency for lightweight tests. By globally configuring and controlling the upper limit of the number of parallel requests per GPU, resource contention and over-subscription in shared scenarios are effectively prevented. This achieves the best balance between improving GPU utilization and ensuring execution stability, making the entire testing process optimal in both resource efficiency and result reliability.
[0064] As an optional embodiment, the large model testing method described above further includes: when testing the target test case, if it is determined that there is an anomaly in the testing process of the target test case, the target test case is retested after the computing resources currently occupied by the target test case are cleared according to the preset retry strategy, and the test count of the target test case is incremented by 1; if it is determined that the test count of the target test case exceeds the maximum retry count, the target test case is marked as a test failure.
[0065] Specifically, during the execution of the target test case, the system monitors its running status in real time. When an abnormal situation is detected in the testing process of the target test case (including but not limited to: task timeout, insufficient video memory, abnormal process exit, etc.), the system automatically triggers the exception handling and retry mechanism. First, according to the preset retry strategy, the computing resources currently occupied by the target test case (including GPU video memory, process context, etc.) are thoroughly cleaned up and released to avoid residual resources interfering with subsequent test execution. After the resource cleanup is completed, the system retests the target test case and increments its test count by 1. Optionally, the number of retryes can be constrained by the preset maximum number of retryes in the global configuration file. Each retry repeats the above process of "resource cleanup → re-execution → count increment". If the system detects an exception again during each retry, it continues to clean up resources and retry again according to the same strategy until the test is successfully completed or the cumulative number of tests exceeds the preset maximum number of retryes. When it is determined that the number of tests for the target test case exceeds the maximum number of retryes, the system stops retrying and marks the target test case as a failed test. Furthermore, it can record the reasons for failure and context information, including the complete log of the failure, input parameters, GPU usage snapshots (memory usage, computing power utilization, etc.), and retry history, which facilitates subsequent troubleshooting and analysis.
[0066] Furthermore, after each target test batch is completed, the system proceeds to the test result aggregation and metric reporting step. First, it aggregates the test logs and results of each process within the target test batch, generating a unified test report containing key information such as the execution status (pass / fail / skipped), time consumption, and resource usage of each target test case. For service performance testing scenarios, the system further extracts key performance indicators such as QPS, P90 / P95 / P99 latency, average latency, and maximum latency from the test results. These indicators are then reported to a unified monitoring platform via monitoring systems such as Prometheus, enabling visualized display of test data and historical trend comparison analysis, providing data support for performance regression testing and capacity planning.
[0067] This implementation effectively reduces the impact of occasional issues such as insufficient GPU memory, process crashes, and timeouts on the stability of the CI pipeline through automated anomaly detection, resource cleanup, and retry mechanisms. It significantly improves the self-healing capability and success rate of the testing process and reduces the cost of manual intervention for troubleshooting. In addition, by recording detailed failure context information (logs, parameters, GPU snapshots, etc.), the difficulty of problem localization and repair is greatly reduced. By extracting and reporting key indicators from service performance testing scenarios in a structured manner, long-term accumulation and visualization of performance data are achieved, enabling timely detection of performance degradation trends and providing quantitative basis for capacity planning and preheating strategy formulation for subsequent production deployment.
[0068] Example 1: MR CI scenario based on multi-GPU server This embodiment uses a server with 8 GPU cards as an example to illustrate the application method in MR CI scenarios.
[0069] 1. Test metadata management: During the project initialization phase, a test metadata management module is built to create a record entry for each test case (i.e., test case). The record fields include, but are not limited to: Case unique identifier (e.g., tests / server / test_tree_server.py::test_xxx); The corresponding business module or configuration file path; The type of model and inference service it relies on (such as single agent server, combined agent server, etc.). The required number of GPUs is gpu_num; The concurrency attribute `concurrency_type` for a case can be either `exclusive` or `shared`. Historical average time (avg_cost_time) and maximum time (max_cost_time); Historical peak memory usage (max_memory_usage); Test type tags (such as unit_cpu, unit_gpu, server_perf, etc.).
[0070] The above information can be automatically collected through regular full-scale testing and stored in a JSON file, database, or configuration center. For newly added test cases, their metadata can be supplemented after the first execution; if the metadata is incomplete, default values can be assigned according to a conservative strategy.
[0071] 2. Incremental Testing Analysis: When MR CI is triggered, the incremental test parsing module reads the code differences of this commit, including: Adding, deleting, and modifying source code files; Configuration file changes (especially server cfg); Third-party dependency version changes.
[0072] When MR CI is triggered, the modification information of the target model is obtained by parsing the code differences of this commit. The modification information includes: file information of the specified module files that have changed (addition, deletion, and modification of source code files), changes to configuration files (especially server cfg), and modification information used to indicate whether the third-party dependencies of the target model have been changed (changes in third-party dependency versions).
[0073] The parsing module first identifies all target test cases corresponding to all changed specified module files based on a pre-determined "module file → test case file / ID" mapping relationship, that is, it finds all directly related test case files; for complex server configurations, it further parses the runners, model weights and toolchains referenced in the configuration file, and then locates the test cases that depend on these components.
[0074] When it is determined, based on the modification information, that the third-party dependencies of the target model have indeed changed, such as an upgrade of a deep learning framework or middleware version, the test cases corresponding to all module files in the target model are identified as all target test cases. That is, through the mapping table of "third-party dependencies → test case files", all potentially affected test cases are identified and added to the target test case set. If the mapping relationship is missing or the scope of impact cannot be determined, the test is degraded to a full test.
[0075] 3. Divide the target test batches according to resource consumption: The target test case set obtained from incremental parsing is denoted as C = {c1, c2, …, c n By obtaining the historical test time and GPU requirements for each target test case, the resource consumption of each target test case is determined. The batch partitioning module partitions the batches based on the following information, according to the GPU requirements of each target test case and the first batch partitioning requirements: The number of GPUs required for each test case is gpu_num(c i ); Historical average time avg_cost_time(c i ); The total number of GPUs on a single machine is GPU_TOTAL (e.g., 8).
[0076] One feasible strategy is: Test cases are categorized by gpu_num, such as 1 card, 2 cards, 4 cards, etc., so that the difference in the number of GPUs required by each test case in the same target test batch is less than the preset difference limit. Within each category (i.e., a target test batch), sort the target test cases in descending order of avg_cost_time; Initialize a fixed number of batches, for example, 8 batches, corresponding to the subsequent 8 Tasks; Iterate through the sorted list in order, assign the current test case to the batch with the "lowest cumulative time" each time, and record the corresponding total gpu_num requirement in the batch for subsequent estimation of Pod resources; For test cases that lack historical execution time records, they are placed in a temporary list called extra, and then allocated to each batch in sequence according to a simple round-robin strategy after the main partition is completed.
[0077] After dividing all target test cases according to the GPU requirement for each target test case and the first batch partitioning requirements to obtain multiple specified test batches, if there is a first test batch among the multiple specified test batches that does not meet the second batch partitioning requirements (i.e., the number of test cases in a single batch exceeds a preset threshold), then the first test batch is split to obtain multiple second test batches that meet the second batch partitioning requirements. Based on the second test batches and the remaining specified test batches other than the first test batches among the multiple specified test batches, multiple target test batches are obtained.
[0078] By employing the above strategies, the resource consumption of each target test batch can be made roughly similar on average, and the total time consumption can be relatively balanced, thereby reducing the overall end-to-end test time when multiple tasks are executed in parallel.
[0079] 4. Request and calculate resources according to the maximum resource usage limit: Based on the batch partitioning results, for each target test batch, the CIPipeline configuration is generated or updated according to the resource consumption limit of that target test batch (i.e., the amount of resources required by the target test case that consumes the most resources in that target test batch). Specifically, the configuration information is used to define the Pod specifications required for each Task (i.e., task) corresponding to each target test batch. The Pod specifications include: number of GPUs, GPU type, CPU resources, and memory resources. The Pod specifications required for each task in each target test batch are greater than or equal to the resource consumption limit of that target test batch. In a typical configuration: The number of Tasks is fixed at 8; Each Task receives a list of test cases to be run in this batch via parameters (e.g., in the form of pytest command-line parameters). Before each Task runs, it calculates the required GPU resources for the Pod based on the maximum gpu_num and number of test cases in the batch. For example, if all test cases in the batch only require 1 GPU, the Pod can be configured with 1 to 2 GPUs to improve concurrency; if there are test cases with 2 or 4 GPUs, the number of GPUs in the Pod will be increased accordingly.
[0080] The CI Pipeline generation module can be implemented in two ways: Before code is committed, the Pipeline configuration file is automatically updated via a local script or a pre-commit hook; After the CI pipeline is launched, the initial Task dynamically generates or updates the Pipeline YAML, which is then applied to subsequent executions.
[0081] This embodiment preferably adopts the second approach to reduce intrusion into the developer's workflow. During the execution of the CI pipeline, the corresponding Pod resources are requested from the Kubernetes cluster one by one as needed, according to the Pod specifications of each target test batch, rather than requesting the maximum GPU resources required for all tests at once.
[0082] 5. GPU Resource-Aware Test Case Scheduling - Parallel Testing Through Computational Resources Within the Pod corresponding to each target test batch, start the test execution script, for example: python -m pytest -n auto --dist=loadfile … Here, `-n auto` indicates that the number of workers is automatically selected based on the number of available CPU cores, and `--dist=loadfile` controls that test cases in the same file are executed in the same process. If the Pod resources corresponding to the target test batch exceed the resource usage limit for that target test batch, multiple test processes are started within that Pod, with each test process executing one test case. A specific GPU visibility set (e.g., setting `CUDA_VISIBLE_DEVICES`) is allocated to each worker when it starts, using monkeypatch or a custom pytest plugin.
[0083] During test execution, the GPU monitoring component (which can be implemented using tools such as nvitop) periodically acquires real-time monitoring data of each GPU card in the Pod resources, including the memory usage and computing power utilization of each GPU card, and exposes this data to the scheduling module. The scheduling module uses prior data such as: the gpu_num of the current test case to be executed and historical memory usage data (such as max_memory_usage); real-time monitoring data such as the current free_memory and utilization of each GPU card; and if the real-time monitoring data determines that there are idle GPUs in the Pod resources (i.e., the memory usage and computing power utilization are lower than the preset load requirements), at least one test process among multiple test processes calls the idle GPU to test the first test case to be executed in the target test batch. The first test case (i.e., the target test case marked with the concurrency attribute as shared and whose GPU can be shared with other target test cases) is dynamically allocated to a GPU with low current load and sufficient memory, thereby achieving load balancing among multiple GPU cards.
[0084] 6. Concurrent Attribute Constraint Execution - Parallel Testing with Computational Resources For target test cases marked as exclusive (i.e., the second test case), the system forces them to run exclusively on the GPU in the computing resources they belong to, that is, they can only run exclusively on a certain GPU at any given time, and other test cases are prohibited from being launched on the same GPU, in order to avoid insufficient video memory or computing power contention leading to test instability.
[0085] For target test cases marked as "shared" (i.e., the first test case), the system tests them by sharing the GPUs within the computing resources. The number of test cases running simultaneously on the same GPU is controlled by the globally configured maximum parallelism per GPU (MAX_SHARED_PER_GPU). For example, when MAX_SHARED_PER_GPU=2, a maximum of two shared test cases are allowed to run concurrently on the same GPU, balancing resource utilization and execution stability.
[0086] 7. Exception Handling and Retry During test execution, if any of the following occurs, the test process of the target test case is determined to be abnormal, and the exception handling module is triggered: the test case exceeds the preset timeout period; the test process throws an uncaught exception; the inference service is forcibly terminated by the operating system due to insufficient video memory or other reasons; the number of failures in the pytest report exceeds the threshold (e.g., --maxfail=5).
[0087] The exception handling module first performs resource cleanup operations on the computing resources currently occupied by the target test case according to the exception type, such as releasing the corresponding GPU binding, terminating residual processes, and cleaning up temporary files. Then, it re-tests the target test case according to the preset retry strategy and increments the test count of the target test case by 1: for exceptions related to insufficient video memory, it can try to reschedule the test case on a GPU with a lower load; for failures caused by occasional network or service fluctuations, it can directly re-execute the test case; if it is determined that the test count of the target test case exceeds the maximum number of retries (i.e., the test case still fails after reaching the maximum number of retries), the target test case is marked as a test failure, and the failure reason and context information (including logs, input parameters, GPU usage snapshots, etc.) are recorded in detail in the final report.
[0088] 8. Summarizing test results and reporting indicators After each target test batch is completed, the test result aggregation module collects test logs and result files from different tasks and merges them into a single overview report. For server performance testing scenarios, key metrics such as QPS, P90 / P95 / P99 latency, average latency, and maximum latency are extracted from the test results and pushed to Prometheus or other monitoring systems by the metric reporting module, enabling visualization of test data and comparison with historical trends.
[0089] Example 2: Resource-constrained scenarios based on shared Kubernetes clusters In a shared Kubernetes cluster, GPU resources are relatively scarce, and there is no dedicated CI machine. This embodiment, based on the method in Embodiment 1, further demonstrates the characteristic of reducing the peak value of a single resource request through batch partitioning.
[0090] In this scenario: 1. The number of tasks is fixed, for example, still 8; 2. Batch partitioning pays more attention to balancing the "total number of GPUs required per batch" to avoid excessive Pod resource requests in a certain batch, resulting in long queues; 3. For test cases requiring 1 GPU, if resources are sufficient, a Pod with 2 to 4 GPUs can be configured for a batch to improve utilization through multi-process concurrent execution within a single Pod; 4. For test cases requiring 2 or more GPUs, try to ensure that the number of high-demand test cases in each batch is relatively even.
[0091] By adopting the above method, without changing the core scheduling logic, this invention can adjust the batch division and Pod specifications according to the cluster resource tension, so that the CI process still has good throughput and stability in a resource-constrained environment.
[0092] Example 3: Support for expansion across multiple CI platforms and hardware form factors This embodiment illustrates the scalability of the invention on different CI platforms and in different hardware environments: 1. When the CI platform is GitLab CI or GitHub Actions, the target test batch partitioning results can be mapped to a Job matrix, and different target test batches can be executed concurrently as different elements in the matrix; 2. When the hardware is CPU-only or other accelerator cards (such as FPGA, dedicated inference chip), the "number of GPUs" and "video memory usage" can be replaced with indicators such as "number of accelerator cards" and "memory usage", and the scheduling logic remains unchanged; 3. When the Agent application is extended to multimodal or world model scenarios, the scheduling module can adapt naturally simply by adding fields such as the corresponding model type and input data scale to the test metadata.
[0093] Furthermore: 1. The batch partitioning algorithm in this invention is not limited to a greedy strategy; dynamic programming, integer programming, or machine learning-based predictive models can also be used to further improve scheduling quality. 2. The GPU monitoring component can be implemented based on nvidia-smi, DCGM, or third-party open-source tools; the specific implementation method does not affect the essence of this invention. 3. The incremental test parsing module can combine static code analysis, runtime coverage statistics, and other methods to further narrow down the set of "affected" test cases. 4. The CI Pipeline generation module can use a template engine (such as Jinja2) combined with configuration files (such as YAML and JSON) to dynamically generate various different Pipeline variants according to project needs. 5. The method and system proposed in this invention can be deployed as an independent test management platform, or some functions can be decentralized to various project repositories for reuse through a unified library or script.
[0094] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.
[0095] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM (Read-Only Memory) / RAM (Random Access Memory), magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0096] According to another aspect of the embodiments of this application, a large model testing apparatus for implementing the above-described large model testing method is also provided. Figure 3 This is a structural block diagram of an optional large model testing apparatus according to an embodiment of this application, such as... Figure 3 As shown, the device may include: Module 31 is used to acquire the target model to be tested; Module 32 is used to determine all target test cases for testing the target model based on the modification information of the target model. The partitioning module 33 is used to divide all target test cases into multiple target test batches according to the resource consumption of each target test case in all target test cases. In the case that there are multiple target test cases in the same target test batch, the difference between the resource consumption of each target test case in the multiple target test cases is less than a preset difference upper limit. Application module 34 is used to apply for computing resources corresponding to each target test batch according to the upper limit of resource consumption of each target test batch. The upper limit of resource consumption of each target test batch is the amount of resources required by the target test case with the highest resource consumption in each target test batch. Test module 35 is used to perform parallel testing on each target test batch using the computing resources corresponding to each target test batch.
[0097] It should be noted that the acquisition module 31 in this embodiment can be used to perform the above step S102, the determination module 32 in this embodiment can be used to perform the above step S104, the division module 33 in this embodiment can be used to perform the above step S106, the application module 34 in this embodiment can be used to perform the above step S108, and the test module 35 in this embodiment can be used to perform the above step S110.
[0098] In addition to the modules described above, the apparatus in this embodiment may also include modules that execute any method in any of the aforementioned large model testing methods.
[0099] It should be noted that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules, as part of the device, can operate in ways such as... Figure 1 The method shown can be implemented in either software or hardware within a hardware environment, where the hardware environment includes a network environment.
[0100] According to another aspect of the embodiments of this application, an electronic device for implementing the above-described large model testing method is also provided. The electronic device may be a server, a terminal, or a combination thereof.
[0101] According to another embodiment of this application, an electronic device is also provided, comprising: Figure 4 As shown, the electronic device may include: a processor 1501, a communication interface 1502, a memory 1503, and a communication bus 1504, wherein the processor 1501, the communication interface 1502, and the memory 1503 communicate with each other through the communication bus 1504.
[0102] Memory 1503 is used to store computer programs; When processor 1501 executes the program stored in memory 1503, it performs the following steps: Step S102: Obtain the target model to be tested.
[0103] Step S104: Based on the modification information of the target model, determine all target test cases for testing the target model.
[0104] Step S106: According to the resource consumption of each target test case in all target test cases, divide all target test cases into multiple target test batches. Where there are multiple target test cases in the same target test batch, the difference between the resource consumption of each target test case in the multiple target test cases is less than the preset difference upper limit.
[0105] Step S108: Apply for computing resources corresponding to each target test batch according to the upper limit of resource consumption for each target test batch. The upper limit of resource consumption for each target test batch is the amount of resources required by the target test case that consumes the most resources in each target test batch.
[0106] Step S110: Perform parallel testing on each target test batch using the computing resources corresponding to each target test batch.
[0107] Optionally, in this embodiment, the communication bus can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used to represent it in the figure, but this does not mean that there is only one bus or one type of bus. The communication interface is used for communication between the aforementioned electronic device and other devices.
[0108] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.
[0109] As an example, the memory 1503 described above may include, but is not limited to, the acquisition module 31, determination module 32, partitioning module 33, application module 34, and testing module 35 from the large model testing device described above. Furthermore, it may include, but is not limited to, other module units from the large model testing device described above, which will not be elaborated upon in this example.
[0110] The processor mentioned above can be a general-purpose processor, including but not limited to: CPU (Central Processing Unit), NP (Network Processor), etc.; it can also be DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0111] This application also provides a computer-readable storage medium, which includes a stored program, wherein the program executes the method steps of the above method embodiments when it runs.
[0112] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, ROMs, RAMs, portable hard drives, magnetic disks, or optical disks.
[0113] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0114] If the integrated units in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.
[0115] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0116] In the several embodiments provided in this application, it should be understood that the disclosed client can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between units or modules, and may be electrical or other forms.
[0117] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the solution provided in this embodiment, depending on actual needs.
[0118] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0119] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.
Claims
1. A large-scale model testing method, characterized in that, include: Obtain the target model to be tested; Based on the modification information of the target model, determine all target test cases for testing the target model; Based on the resource consumption of each target test case in all target test cases, all target test cases are divided into multiple target test batches. In the case where multiple target test cases are contained in the same target test batch, the difference between the resource consumption of each target test case in the multiple target test cases is less than a preset difference upper limit. According to the upper limit of resource consumption for each target test batch, the computing resources corresponding to each target test batch are applied for, wherein the upper limit of resource consumption for each target test batch is the amount of resources required by the target test case that consumes the most resources in each target test batch; Each target test batch is tested in parallel using the computing resources corresponding to each target test batch.
2. The large model testing method according to claim 1, characterized in that, Based on the modification information of the target model, all target test cases for testing the target model are determined, including: The modification information is obtained by parsing the specified module file that has been modified in the target model, wherein the modification information includes: file information of the specified module file that has been modified; Based on the modified information and the pre-determined mapping relationship between module files and test cases, all target test cases corresponding to all the specified module files are determined.
3. The large model testing method according to claim 1, characterized in that, Based on the modification information of the target model, all target test cases for testing the target model are determined, including: The modification information is obtained by analyzing whether the third-party dependencies of the target model have changed relative to the third-party dependencies of the previous version of the target model. The modification information includes: modification information used to indicate whether the third-party dependencies of the target model have changed. If, based on the modified information, it is determined that the third-party dependencies of the target model have changed relative to the third-party dependencies of the previous version of the target model, then the test cases corresponding to all module files in the target model are determined as all target test cases.
4. The large model testing method according to claim 1, characterized in that, The step involves dividing all target test cases into multiple target test batches based on the resource consumption of each target test case, including: By obtaining the historical test time and GPU requirements for each target test case, the resource consumption of each target test case is determined. According to the GPU requirement of each target test case and the first batch division requirements, all target test cases are divided to obtain multiple target test batches. The first batch division requirements include: when the same target test batch contains multiple target test cases, the difference between the GPU requirements of each target test case in the multiple target test cases is less than a preset difference upper limit.
5. The large model testing method according to claim 4, characterized in that, The process involves dividing all target test cases according to the GPU requirement of each target test case and the first batch division requirements to obtain the multiple target test batches, including: All target test cases are divided according to the GPU requirement for each target test case and the first batch division requirements to obtain multiple specified test batches; If, among the multiple specified test batches, there is a first test batch that does not meet the second batch division requirements, the first test batch is split to obtain multiple second test batches that meet the second batch division requirements. The second batch division requirements include: the number of test cases contained in a test batch is less than or equal to a preset threshold. The plurality of target test batches are obtained based on the second test batch and the remaining specified test batches other than the first test batch among the plurality of specified test batches.
6. The large model testing method according to claim 1, characterized in that, The process of requesting computing resources corresponding to each target test batch according to the maximum resource consumption of each target test batch includes: Based on the resource consumption limit of each target test batch, configuration information corresponding to all target test batches is generated. The configuration information is used to define: the Pod specification required to execute the task of each target test batch. The Pod specification required for the task of each target test batch is greater than or equal to the resource consumption limit of each target test batch. The Pod specification includes: number of GPUs, GPU type, CPU resources, and memory resources. According to the Pod specifications of each target test batch, request the Pod resources corresponding to each target test batch as the computing resources corresponding to each target test batch.
7. The large model testing method according to claim 6, characterized in that, The step of performing parallel testing on each target test batch using the computing resources corresponding to each target test batch includes: If the Pod resources corresponding to the target test batch are greater than the upper limit of the resource usage of the target test batch, multiple test processes are started within the Pod corresponding to the target test batch, wherein each test process executes one test case. The system acquires real-time monitoring data of the Pod resources, including the memory usage and computing power utilization of each GPU in the Pod resources. If, based on the real-time monitoring data, it is determined that there is an idle GPU in the Pod resources, at least one of the multiple test processes calls the idle GPU to test the first test case to be executed in the target test batch. The idle GPU is defined as one whose video memory usage and computing power utilization are lower than the preset load requirement, and the first test case is a target test case whose GPU can be shared with other target test cases.
8. The large model testing method according to claim 1, characterized in that, The step of performing parallel testing on each target test batch using the computing resources corresponding to each target test batch includes: If the target test case in the target test batch is the second test case, the second test case is tested by using the GPU to which the second test case belongs in the computing resources exclusively. If the target test case in the target test batch is the first test case, the first test case is tested by sharing the GPU to which the first test case belongs in the computing resources.
9. The large model testing method according to claim 1, characterized in that, The method further includes: If an anomaly is detected during the testing of the target test case, the computing resources currently occupied by the target test case are cleared according to the preset retry strategy, the target test case is retested, and the test count of the target test case is incremented by 1. If the number of times the target test case is tested exceeds the maximum number of retries, the target test case is marked as a test failure.
10. A large-scale model testing device, characterized in that, include: The acquisition module is used to acquire the target model to be tested; The determination module is used to determine all target test cases for testing the target model based on the modification information of the target model. The partitioning module is used to divide all target test cases into multiple target test batches according to the resource consumption of each target test case in all target test cases. In the case that there are multiple target test cases in the same target test batch, the difference between the resource consumption of each target test case in the multiple target test cases is less than a preset difference upper limit. The application module is used to apply for computing resources corresponding to each target test batch according to the upper limit of resource consumption for each target test batch, wherein the upper limit of resource consumption for each target test batch is the amount of resources required by the target test case that consumes the most resources in each target test batch; The testing module is used to perform parallel testing on each target test batch using the computing resources corresponding to each target test batch.
11. An electronic device comprising a processor, a communication interface, a memory, and a communication bus, wherein, The processor, the communication interface, and the memory communicate with each other via the communication bus, characterized in that... The memory is used to store computer programs; The processor is configured to perform the method of any one of claims 1 to 9 by running the computer program stored in the memory.
12. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to execute the method described in any one of claims 1 to 9 when it is run.
