Model testing system and method based on multi-chip computing equipment and related equipment

Abstract

The invention provides a model testing system and method for multi-chip computing equipment and related equipment. The system comprises an input module used for obtaining command line parameters and configuration parameters; the main control module is used for analyzing the command line parameter and the configuration parameter and sending a reasoning command to the performance pressure measurement module; the main control module is used for analyzing command line parameters, the API service starting module is used for starting reasoning service based on the analyzed command line parameters, and the performance pressure testing module is used for constructing a plurality of parameter combinations based on the analyzed configuration parameters, carrying out reasoning testing based on each parameter combination and sending corresponding reasoning results to the main control module; and the output module is used for outputting reasoning results corresponding to the parameter combinations. During testing, a user only needs to input command line parameters and configuration parameters, and the main control module can schedule the API service starting module to configure environment variables, load a to-be-detected model, send a reasoning request to an API reasoning service process based on each parameter combination and start reasoning.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of deep learning, and in particular to a model testing system and method based on a multi-chip computing device and related equipment. BACKGROUND

[0002] Before being deployed to special hardware, a large model (such as a large language model) usually needs to be performance stress tested and verified. At present, when a large model is performance stress tested and verified, a manual testing method is usually used. When manually testing, an operator needs to manually build a test environment, configure test parameters, etc. In addition, when manually testing, after testing a set of parameters, the test parameters need to be reconfigured to perform the next test. This manual testing method is not only prone to errors but also inefficient. SUMMARY

[0003] The main purpose of the embodiments of the present application is to provide a model testing system and method based on a multi-chip computing device and related equipment, aiming to solve the problem that manual testing is prone to errors and inefficient.

[0004] The present application provides a model testing system based on a multi-chip computing device, comprising an input module, a main control module, an API service starting module, a performance stress testing module, and an output module. The input module is configured to obtain command line parameters and configuration parameters. The command line parameters represent a target chip number, a target port, a to-be-tested model path, and a target inference framework path. The configuration parameters include an input token length, an output token length, a concurrency number, a sample number, and a number of repeated tests. The main control module is configured to parse the command line parameters and the configuration parameters, send a starting command to the API service starting module based on the parsed command line parameters, and send an inference command to the performance stress testing module based on the parsed configuration parameters in response to receiving starting success information sent by the API service starting module. The API service starting module is configured to start an inference service based on the parsed command line parameters and send starting success information to the main control module. The performance stress testing module is configured to receive the inference command, construct a plurality of parameter combinations based on the parsed configuration parameters, perform inference testing based on each parameter combination, respectively, and send an inference result corresponding to each parameter combination to the main control module. The plurality of parameter combinations include all combinations of the configuration parameters. The main control module is further configured to receive the inference result corresponding to each parameter combination and send the inference result corresponding to each parameter combination to the output module. The output module is configured to output the inference result corresponding to each parameter combination.

[0005] In the embodiments of the present application, before the inference result corresponding to each parameter combination is sent to the output module, the main control module is further configured to perform structured CSV format processing on the inference result corresponding to each parameter combination.

[0006] In the embodiments of the present application, the system comprises an environment test module, and the main control module is further configured to send an environment test command to the environment test module. The environment test module is configured to detect the hardware connection state, the drive state, the communication link state of the multi-chip computing device, and the installation of the target inference framework on the multi-chip computing device.

[0007] In the embodiments of the present application, the environment test module comprises: a drive detection unit configured to detect or install required hardware drivers; a communication link detection unit configured to detect the communication state of RDMA or DMA in the multi-chip computing device; a computing unit detection unit configured to verify whether the matrix computing function of the multi-chip computing device is normal; an inference framework dependency detection unit configured to verify the integrity and compatibility of the installation of the target inference framework.

[0008] In the embodiments of the present application, the test result corresponding to each parameter combination comprises: request throughput, token throughput, first token delay, average remaining token delay, and total inference time.

[0009] In the embodiments of the present application, the main control module is further configured to aggregate the inference results corresponding to each parameter combination into a single file.

[0010] The present application also proposes a model test method based on a multi-chip computing device, which is applied to the model test system based on a multi-chip computing device as described in any one of the above embodiments, and the method comprises: based on the main control module obtaining command line parameters and configuration parameters from the input module, the command line parameters indicating target chip numbers, target ports, paths of models to be tested, and paths of target inference frameworks, and the configuration parameters comprising input token length, output token length, number of concurrences, number of samples, and number of repeated tests; based on the main control module analyzing the command line parameters and the configuration parameters; based on the main control module and the analyzed command line parameters, sending a start command to the API service start module; In response to receiving the start success information sent by the API service start module, the main control module sends an inference command to the performance stress testing module based on the main control module and the parsed configuration parameters. The main control module receives the inference result corresponding to each parameter combination sent by the performance stress testing module, and sends the inference result corresponding to each parameter combination to the output module. Each parameter combination is obtained based on the performance stress testing module traversing the parsed configuration parameters, and the inference result corresponding to any parameter combination is obtained based on the performance stress testing module inferring with the parameter combination as a test condition.

[0011] In the embodiments of the present application, before sending the inference result corresponding to each parameter combination to the output module, the method further comprises: The main control module respectively performs structured CSV format processing on the inference result corresponding to each parameter combination.

[0012] In the embodiments of the present application, the method further comprises: In response to receiving the environment test command sent by the main control module, the environment test module detects the hardware connection state, the drive state, the communication link state of the multi-chip computing device, and the installation of the target inference framework on the multi-chip computing device.

[0013] In the embodiments of the present application, the detection of the hardware connection state, the drive state, the communication link state of the multi-chip computing device, and the installation of the target inference framework on the multi-chip computing device based on the environment test module comprises: Detecting or installing the hardware driver required by the drive detection unit; Detecting the communication state of RDMA or DMA in the multi-chip computing device based on the communication link detection unit; Verifying whether the matrix calculation function of the multi-chip computing device is normal based on the calculation unit detection unit; Verifying the integrity and compatibility of the installation of the target inference framework based on the inference framework dependency detection unit.

[0014] In the embodiments of the present application, the test result corresponding to each parameter combination comprises: request throughput, token throughput, first token delay, average token delay, and total inference time.

[0015] In the embodiments of the present application, the method further comprises: based on the output module, the inference result corresponding to each parameter combination is summarized into a single file.

[0016] The present application also proposes a computer readable storage medium comprising instructions which, when executed on a computer, cause the computer to perform the method of any one of the above.

[0017] The application also provides a computing device, comprising a processor, and a computer program stored on the processor, wherein the computer program is configured to implement the method according to any one of the preceding embodiments.

[0018] In the embodiment of the application, when testing the to-be-detected model, the user only needs to input command line parameters and configuration parameters based on the input module, and the main control module can automatically configure environment variables, load the to-be-detected model and start the API inference service process by scheduling the API service start module. In addition, the main control module can also schedule the performance stress testing module to traverse the configuration parameters to obtain various parameter combinations, and respectively send inference requests to the API inference service process based on each parameter combination, and the main control module can also output the inference results corresponding to each parameter combination through the output module for the user to view. The entire testing process has a high degree of automation, is less prone to errors, and can obtain the test results under all parameter combinations at one time, and has a high testing efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0019] The purposes, features and advantages of the embodiments of the application will become apparent from the detailed description of the embodiments of the application with reference to the accompanying drawings. Among them: Figure 1 a module diagram of the model testing system based on a multi-chip computing device in an embodiment of the application; Figure 2 a step diagram of the model testing method based on a multi-chip computing device in an embodiment of the application; Figure 3 a module diagram of the computer readable storage medium in an embodiment of the application; Figure 4 a module diagram of the computing device in an embodiment of the application.

[0020] The purposes, features and advantages of the embodiments of the application will become apparent from the detailed description of the embodiments of the application with reference to the accompanying drawings. Among them: DETAILED DESCRIPTION

[0021] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the application. Obviously, the described embodiments are only some of the embodiments of the application, but not all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the application.

[0022] The flowcharts shown in the drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor do they have to be executed in the order described. For example, some operations / steps can be further divided, combined or partially merged, so the actual execution order may be changed according to the actual situation.

[0023] It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in this specification and the appended claims, the singular forms "a," "an" and "the" include plural referents unless the context clearly dictates otherwise.

[0024] Exemplary apparatus As Figure 1 shown, the embodiment of the present application provides a model test system based on multi-chip computing device, comprising: an input module 110, a main control module 120, an API service starting module 130, a performance stress test module 140, an output module 150; The input module 110 is configured to obtain command line parameters and configuration parameters, wherein the command line parameters represent target chip number, target port, to-be-tested model path, and target inference framework path, and the configuration parameters include input token length, output token length, concurrency number, sample quantity, and repeated test number. The main control module 120 is configured to parse the command line parameters and the configuration parameters, send a starting command to the API service starting module 130 based on the parsed command line parameters, and send an inference command to the performance stress test module 140 based on the parsed configuration parameters in response to receiving starting success information sent by the API service starting module 130. The API service starting module 130 is configured to start an inference service based on the parsed command line parameters, and send starting success information to the main control module 120. The performance stress test module 140 is configured to receive the inference command, construct a plurality of parameter combinations based on the parsed configuration parameters, perform inference test based on each parameter combination respectively, and send an inference result corresponding to each parameter combination to the main control module 120, wherein the plurality of parameter combinations include all combination conditions in the configuration parameters. The main control module 120 is further configured to receive the inference result corresponding to each parameter combination, and send the inference result corresponding to each parameter combination to the output module 150. The output module 150 is configured to output the inference result corresponding to each parameter combination.

[0025] In the embodiment of the present application, when a to-be-tested model is tested based on a multi-chip computing device, the to-be-tested model needs to be carried on a required target inference framework, and the computing capability of the multi-chip computing device is used to perform inference test on the to-be-tested model.

[0026] In the embodiments of the present application, the multi-chip computing device can be a hardware computing device or a server integrating multiple parallel processing chips, which provides aggregated computing power through distributed computing to accelerate large model inference tasks. Among them, multiple parallel processing chips can be connected through high-speed interconnection technology (such as NVLink, NVSwitch, InfiniBand) to realize low-latency and high-bandwidth data exchange. For example, the multi-chip computing device can be: Multi-card GPU server: such as a server equipped with 8* NVIDIA A100 / P100.

[0027] TPU Pod: a computing cluster composed of multiple TPUs.

[0028] In the embodiments of the present application, the chip number is a logical identifier assigned to each parallel processing chip in the operating system of the multi-chip computing device, which is used to uniquely specify and address a specific processing chip at the software level. For example: In a Linux operating system composed of 4 parallel processing chips with 4 GPUs, the 4 GPUs can be numbered 0, 1, 2, and 3. The target chip number is the number of the processing chip that needs to be used when testing the to-be-tested model, which can be represented by the environment variable CUDA_VISIBLE_DEVICES=“0,1”, indicating that the to-be-tested model needs to use two GPUs numbered 0 and 1 when testing.

[0029] In the embodiments of the present application, the inference framework is a software library or running system for executing large model forward propagation. The inference framework provides model loading, computation optimization, hardware acceleration, request scheduling, etc. The inference framework can convert static model files into serviceable APIs. Common inference frameworks are, for example: vLLM, TensorRT, ONNX Runtime, TritonInference Server. The target inference framework is the inference framework required by the to-be-tested model when testing.

[0030] In the embodiments of the present application, the to-be-tested model path is the path of the directory storing the weight file and configuration file of the to-be-tested model.

[0031] In the embodiments of the present application, the port is a logical endpoint in network communication, identified by a 16-bit unsigned integer (range 0-65535), and the port, combined with the IP address, can be used to uniquely identify a specific application process on a host. In the embodiments of the present application, the target port is the network entry that the inference service listens to, through which a connection is established with the service API of the to-be-tested model and inference requests are sent.

[0032] In the embodiments of the present application, the token is a basic unit in the model vocabulary table, which is segmented by the tokenizer from the original input text, and the input token length is the maximum number of tokens that can be processed by the model to be tested at a time.

[0033] In the embodiments of the present application, the output token length is the maximum number of tokens generated by the model to be tested at a time when generating text. For example, if the output token length is 50, it means that the model to be tested can only generate text containing 50 tokens at a time at most.

[0034] In the embodiments of the present application, the concurrency number represents the number of independent requests sent to the model to be tested at the same time, which is used to simulate the scenario of multiple users accessing at the same time. For example, if the concurrency number is 32, it means that 32 users send inference requests to the model to be tested at the same time.

[0035] In the embodiments of the present application, the sample number is the total number of samples input when testing the model to be tested.

[0036] In the embodiments of the present application, the input module 110 can provide an interactive interface for the user to input test parameters or issue instructions to the operating system or the model to be tested. Among them, the target chip number, the target port, the path of the model to be tested, and the path of the target inference framework can be input through the command line of the input module 110, and the input token length, the output token length, the concurrency number, the sample number, and the number of repeated tests can be input through the configuration file to form configuration parameters.

[0037] In the embodiments of the present application, the main control module 120 can be deployed based on a Python script file, such as run_all.py. The main control module 120 can automatically read the user's input from the input module 110, such as reading the parameters from the command line, such as model / path / to / model, or reading the configuration parameters of the configuration file (input_arg.csv). After reading the command line parameters and the configuration file, the main control module 120 can parse them to obtain the target chip number, the target port, the path of the model to be tested, the path of the target inference framework, the input token length, the output token length, the concurrency number, the sample number, and the number of repeated tests.

[0038] In addition, after parsing the test information, the main control module 120 can schedule the API service startup module 130 to start the inference service. When scheduling the API service startup module 130, the main control module 120 can send the parsed command line parameters to the API service startup module 130.

[0039] In the embodiments of the present application, the API service starting module 130 can be deployed based on a Python script file, such as api_server.py. According to the command line parameters sent by the main control module 120, the API service starting module 130 can automatically convert the static model file of the to-be-tested model stored on the system file into a dynamic model inference service API that can be accessed through the network.

[0040] Specifically, when the API service starting module 130 receives the starting command of the main control module 120, first, the environment variable (CUDA_VISIBLE_DEVICES) is configured according to the target chip number, so as to lock the inference task of the to-be-tested model on the specified processing chip, thereby realizing resource isolation and multi-task parallel testing.

[0041] Then, the API service starting module 130 can splice the various test information received from the main control module 120 to form a starting parameter. For example, the starting parameter includes the target inference framework path, the to-be-tested model path, and the target port. When the starting command is executed based on each starting parameter, the to-be-tested model needs time to be loaded to the target inference framework, and it will not be started immediately. Therefore, the API service starting module 130 will continuously poll and send a starting request based on the target port until the to-be-tested model is successfully loaded and the inference service is ready. At this time, the API service starting module 130 sends the starting success information to the main control module 120.

[0042] When the main control module 120 receives the starting success information sent by the API service starting module 130, it sends an inference command to the performance stress testing module 140 based on the parsed configuration parameters.

[0043] In the embodiments of the present application, the performance stress testing module 140 can be deployed based on a Python script file, such as benchmark.py. When it receives the parsed configuration parameters from the main control module 120, it first traverses the parsed configuration parameters to obtain all parameter combinations.

[0044] For example, in the embodiments of the present application, it is assumed that the parsed configuration parameters are as follows: Input token length: 512 and 1024 Output token length: 128 and 512 Concurrent number: 4 and 8 Sample number: 100 Repeated test number: 2 Then, based on the above configuration parameters, all parameter combinations are obtained, as follows: Parameter combination 1: input token length 512, output token length 128, concurrent number 4, sample number 100, and repeated test number 2; Parameter combination 2: input token length 512, output token length 128, concurrency 8, sample number 100, and repeated test number 2; Parameter combination 3: input token length 512, output token length 512, concurrency 4, sample number 100, and repeated test number 2; Parameter combination 4: input token length 512, output token length 512, concurrency 8, sample number 100, and repeated test number 2; Parameter combination 5: input token length 1024, output token length 128, concurrency 4, sample number 100, and repeated test number 2; Parameter combination 6: input token length 1024, output token length 128, concurrency 8, sample number 100, and repeated test number 2; Parameter combination 7: input token length 1024, output token length 512, concurrency 4, sample number 100, and repeated test number 2; Parameter combination 8: input token length 1024, output token length 512, concurrency 8, sample number 100, and repeated test number 2.

[0045] That is, after traversing the configuration parameters, the above-mentioned eight parameter combinations can be obtained. Since the repeated test number is 2, that is, each parameter combination is tested twice, that is, 16 times of inference tests are required.

[0046] In the embodiment of the present application, after determining all parameter combinations, the performance stress testing module 140 can configure test parameters based on each parameter combination, and send a test request to the started API inference service process based on each parameter combination.

[0047] Taking the above-mentioned parameter combination 1 as an example, when the performance stress testing module 140 sends a test request to the API inference service process, 100 (sample number) texts with a length of 1024 (input token length) are constructed, and requests are sent to 4 (concurrency) threads at the same time, and the model to be tested is required to generate a text with a length of 514 (output token length) for each request.

[0048] In addition, in the embodiment of the present application, when performing inference tests based on each parameter combination, the performance stress testing module 140 also respectively monitors the test results corresponding to each parameter combination. The test results can include the following indexes: request throughput, token throughput, first token delay, average delay of remaining tokens, and total inference time.

[0049] The performance stress testing module 140 sends the test result corresponding to each parameter combination to the main control module 120 respectively after obtaining the test result corresponding to each parameter combination.

[0050] In the embodiment of the present application, the main control module 120 also sends the test result to the output module 150 for output by the output module 150 after obtaining the test result corresponding to each parameter combination.

[0051] In the embodiment of the present application, the output module 150 can be an interface for user interaction to facilitate the user to view the test result.

[0052] In the embodiment of the present application, when testing the model, the user only needs to input the command line parameters and configuration parameters based on the input module 110, and the main control module 120 can schedule the API service startup module 130 to automatically configure the environment variable, load the model to be detected and start the API inference service. In addition, the main control module 120 can also schedule the performance stress testing module 140 to traverse the configuration parameters to obtain each parameter combination, and send an inference request to the API inference service process based on each parameter combination respectively, and the main control module 120 can also output the inference result corresponding to each parameter combination through the output module 150 for the user to view. The whole testing process has a high degree of automation and is less prone to errors, and the test result under all parameter combinations can be obtained at one time, and the test efficiency is high.

[0053] In the embodiment of the present application, before sending the inference result corresponding to each parameter combination to the output module 150, the main control module 120 is also used for structuring CSV format processing of the inference result corresponding to each parameter combination.

[0054] After the main control module 120 receives the inference result corresponding to each parameter combination, the inference result can be reorganized and calculated by using an automatic script, and output in a Comma-Separated Values (CSV) format, which can facilitate the visualization and subsequent analysis of the inference result.

[0055] In the embodiment of the present application, the system comprises an environment testing module 160, and the main control module 120 is also used for sending an environment testing command to the environment testing module 160. The environment testing module is used for detecting the hardware connection state, the drive state, the communication link state of the multi-chip computing device, and the installation situation of the target inference framework on the multi-chip computing device.

[0056] In the embodiments of the present application, the environment test module 160 can be deployed based on a Python script file, such as check_env.py. Before the test starts, the main control module 120 can send an environment test command to the environment test module 160. After receiving the environment test command, the environment test module 160 can automatically detect the hardware connection state, the driver state, the communication link state of the multi-chip computing device, and the installation state of the target inference framework on the multi-chip computing device, to ensure that the subsequent test can proceed smoothly.

[0057] In the embodiments of the present application, the environment test module includes: a driver detection unit for detecting or installing required hardware drivers; a communication link detection unit for detecting the communication state of RDMA or DMA in the multi-chip computing device; a computing unit detection unit for verifying whether the matrix computing function of the multi-chip computing device is normal; an inference framework dependency detection unit for verifying the integrity and compatibility of the installation of the target inference framework.

[0058] The driver detection unit can be deployed based on a Python script file, such as check_dlc_driver, for checking whether the DLC driver exists and is loaded. If the driver is installed, the installation script can be automatically called.

[0059] The general link detection unit can be deployed based on a Python script file, such as check_rdma and check_matmu. Check_rdma is used to check whether the RDMA is normal. If it fails, it can be automatically initialized. Check_dma is used to check whether the DMA module is executable. If it fails, it can be automatically initialized.

[0060] When verifying the integrity and compatibility of the installation of the target inference framework, the inference framework detection unit can try to import the corresponding package in the Python environment, and run a simple model loading or computing operation to verify whether it can normally call the underlying hardware.

[0061] In the embodiments of the present application, the output module 150 is further used to combine the inference results corresponding to each parameter group into a single file.

[0062] In the embodiments of the present application, after each parameter combination test is completed, the main control module 120 receives the indicators contained in the current inference result, and the current inference result can be loaded into a structured CSV result file. Until all parameter combination inference results are completed, taking the above 8 parameter combinations as an example, each parameter combination is tested twice, and the CSV result file will contain the inference results of all 16 tests. In this way, subsequent sorting, filtering, and chart analysis can be performed using Excel or Python.

[0063] Exemplary method As shown in Figure 2 The present exemplary embodiment also proposes a model testing method based on a multi-chip computing device, which is applied to the model testing system based on a multi-chip computing device described in any of the above embodiments. The method comprises the following steps S100-S500: Step S100: Based on the main control module, the command line parameters and the configuration parameters are obtained from the input module. The command line parameters represent the target chip number, the target port, the path of the model to be tested, and the path of the target inference framework. The configuration parameters include the input token length, the output token length, the number of concurrent samples, and the number of repeated tests. Step S200: Based on the main control module, the command line parameters and the configuration parameters are parsed. Step S300: Based on the main control module and the parsed command line parameters, a start command is sent to the API service start module. Step S400: In response to receiving the start success information sent by the API service start module, based on the main control module and the parsed configuration parameters, an inference command is sent to the performance stress testing module. Step S500: Receive the inference results corresponding to each parameter combination sent by the performance stress testing module, and send the inference results corresponding to each parameter combination to the output module. Each parameter combination is obtained based on the performance stress testing module parsing the configuration parameters. The inference result corresponding to any parameter combination is obtained based on the performance stress testing module using the parameter combination as the test condition.

[0064] In the embodiments of the present application, before sending the inference results corresponding to each parameter combination to the output module, the method further comprises: Based on the main control module, the inference results corresponding to each parameter combination are processed in a structured CSV format.

[0065] In the embodiment of the present application, the method further comprises: in response to receiving the environment test command sent by the main control module, detecting the hardware connection state, the drive state, the communication link state of the multi-chip computing device and the installation of the target inference framework on the multi-chip computing device based on the environment test module.

[0066] In the embodiment of the present application, the detection of the hardware connection state, the drive state, the communication link state of the multi-chip computing device and the installation of the target inference framework on the multi-chip computing device based on the environment test module comprises: detecting or installing the hardware drive required by the drive detection unit; detecting the communication state of RDMA or DMA in the multi-chip computing device based on the communication link detection unit; verifying whether the matrix calculation function of the multi-chip computing device is normal based on the calculation unit detection unit; verifying the integrity and compatibility of the installation of the target inference framework based on the inference framework dependency detection unit.

[0067] In the embodiment of the present application, the test result corresponding to each parameter combination comprises: the request throughput, the token throughput, the first token delay, the average token delay of the remaining tokens and the total inference time.

[0068] In the embodiment of the present application, the method further comprises: based on the output module, the inference results corresponding to each parameter combination are summarized into a single file.

[0069] In the test method in the embodiment of the present application, when testing the model, the user only needs to input the command line parameters and configuration parameters based on the input module, the main control module can schedule the API service startup module to automatically configure the environment variable, load the model to be detected and start the API inference service process; in addition, the main control module can also schedule the performance stress testing module to traverse the configuration parameters to obtain each parameter combination, and respectively send an inference request to the API inference service process based on each parameter combination, and the main control module can also output the inference results corresponding to each parameter combination through the output module for the user to view. The whole test process has a high degree of automation and is not prone to errors, and all parameter combinations can be obtained at one time, and the test efficiency is high.

[0070] Exemplary medium After introducing the method, medium and system of the exemplary embodiments of the present application, next, the computer readable storage medium of the exemplary embodiments of the present application is described with reference to the accompanying drawings. Figure 3 The computer readable storage medium of the exemplary embodiments of the present application is described with reference to the accompanying drawings. Figure 3The computer readable storage medium shown is an optical disc 70, on which a computer program (i.e., a program product) is stored, which, when run by a processor, implements each step described in the above method embodiments, for example, based on the main control module obtaining command line parameters and configuration parameters from the input module, the command line parameters indicating a target chip number, a target port, a to-be-tested model path, and a target inference framework, and the configuration parameters including an input token length, an output token length, a concurrency number, a sample number, and a repeated test number; based on the main control module parsing the command line parameters and the configuration parameters; based on the main control module and the parsed command line parameters, sending a start command to the API service start module; in response to receiving the start success information sent by the API service start module, based on the main control module and the parsed configuration parameters, sending an inference command to the performance stress testing module; receiving the inference result corresponding to each parameter combination sent by the performance stress testing module, and sending the inference result corresponding to each parameter combination to the output module; each parameter combination is obtained based on the performance stress testing module traversing the parsed configuration parameters, and the inference result corresponding to any parameter combination is obtained based on the performance stress testing module inferring with the parameter combination as a test condition. The specific implementation of each step is not repeated here. It should be noted that examples of the computer readable storage medium can also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory, or other optical or magnetic storage media, which are not repeated here.

[0071] Exemplary computing device After introducing the method, system and medium of the exemplary embodiments of the present application, next, with reference to Figure 4 to the computing device of the exemplary embodiments of the present application.

[0072] Figure 4 A block diagram of an exemplary computing device 80 suitable for implementing embodiments of the present application is shown, which can be a computer system or a server. Figure 4 The displayed computing device 80 is only an example and should not bring any limitation to the function and use range of the embodiments of the present application.

[0073] As Figure 4As shown, the components of the computing device 80 can include, but are not limited to, one or more processors or processing units 801, a system memory 802, and a bus 803 that couples various system components including the system memory 802 to the processing unit 801.

[0074] The computing device 80 typically includes a variety of computer system readable media. Such media can be any available media that is located either

[0075] The system memory 802 can include computer system readable media in the form of volatile memory, such as random access memory (RAM) 8021 and / or cache memory 8022. The computing device 80 can further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a ROM 8023 can be used to read-only memory, such as a magnetic disk drive (not shown) that reads from or writes to a nonremovable, nonvolatile magnetic media (not shown). Figure 4 Although not specifically shown, a magnetic disk drive can also be coupled to the bus 803 through a data media interface. As will be further Figure 4 discussed below, a data media interface can be any of a group of computer system readable media adapters that are used to facilitate exchange of data between the bus 803 and one or more external devices for transferring data to / from the computing device 80. Examples of data media interfaces

[0076] A program / utility 8025, having a set (at least one) of program modules 8024, can be stored in system memory 802 by way of example, and can include an operating system, one or more application programs, other program modules, and program data, each of which or a combination can include implementation of a network environment. The program modules 8024 generally carry out the functions and / or methodologies of embodiments of the application as described herein.

[0077] The computing device 80 can also communicate with one or more external devices 804 such as a keyboard or a pointing device, through an input / output (I / O) interface. Further, the computing device 80 can communicate with one or more networks, such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet, through a network adapter 806. As will be appreciated, the network adapter 806 can include any Figure 4 of a wide array of well-known devices for allowing communication between the computing device 80 and any such network. As will be appreciated, it will be appreciated that the network adapter 806 can be considered to be part of the communication device 80. While the application has been described in connection with various embodiments, it will be understood that the application is capable of further modifications. Other embodiments will be apparent to those of ordinary skill in the art from consideration of the specification and practice of the application throughout the various embodiments and Figure 4Other hardware and / or software modules can be used in conjunction with the computing device 80, as shown, without departing from the scope of the application.

[0078] The processing unit 801 executes various functional applications and data processing by running programs stored in the system memory 802, for example, based on the main control module obtaining command line parameters and configuration parameters from the input module, the command line parameters representing target chip number, target port, model path to be tested, target inference framework path, the configuration parameters including input token length, output token length, number of concurrency and sample quantity, number of repeated tests; Based on the main control module parsing the command line parameters and the configuration parameters; Based on the main control module and the parsed command line parameters, a start command is sent to the API service start module; In response to receiving the start success information sent by the API service start module, based on the main control module and the parsed configuration parameters, an inference command is sent to the performance stress testing module; The inference results corresponding to each parameter combination are received from the performance stress testing module, and the inference results corresponding to each parameter combination are sent to the output module; wherein each parameter combination is obtained based on the performance stress testing module traversing the parsed configuration parameters, and the inference result corresponding to any parameter combination is obtained based on the performance stress testing module inferring with the parameter combination as the test condition. The specific implementation of each step will not be repeated here. It should be noted that although several units / modules or sub-units / sub-modules of the operation device are mentioned in the foregoing detailed description, such division is only exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided into several units / modules.

[0079] In the description of the present application, it should be noted that the terms "first", "second", "third" are only for the purpose of description, and cannot be understood or implied as indicating or implying relative importance.

[0080] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system, system and unit can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0081] In several embodiments provided in the present application, it should be understood that the disclosed system, system and method can be implemented in other manners. The above described system embodiments are merely illustrative, for example, the division of the units is only a logical function division, and there can be another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between different units, or the among different units, can be indirect couplings or communication connections through some interfaces, and can be in electrical, mechanical or other forms.

[0082] The units described as separated components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purposes of the embodiments of the present application.

[0083] In addition, each functional unit in the embodiments of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit.

[0084] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a non-volatile computer readable storage medium executable by a processor. Based on such understanding, the technical solutions of the present application essentially or the part of the prior art that contributes to the technical solutions or the part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computing device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0085] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present application, and are not intended to limit the technical solutions of the present application. The protection scope of the present application is not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those skilled in the art that any person skilled in the art can make modifications or easy changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed by the present application, or can easily think of changes or equivalent replacements for some of the technical features. The modifications, changes or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0086] In addition, although the operations of the methods of the present application are described in a particular order in the drawings, this does not require or imply that the operations must be performed in that particular order, or that all of the illustrated operations must be performed to achieve the desired results. Additionally or alternatively, certain steps can be omitted, combined into a single step, and / or divided into multiple steps.

Claims

1. A model testing system based on a multi-chip computing device, comprising: Input module, main control module, API service startup module, performance stress testing module, output module; The input module is used to obtain command line parameters and configuration parameters. The command line parameters represent the target chip number, target port, path of the model to be tested, and path of the target inference framework. The configuration parameters include the input token length, output token length, number of concurrent connections, number of samples, and number of repeated tests. The main control module is used to parse the command line parameters and the configuration parameters; and to send a startup command to the API service startup module based on the parsed command line parameters; and in response to receiving a startup success message from the API service startup module, to send an inference command to the performance stress testing module based on the parsed configuration parameters. The API service startup module is used to start the inference service based on the parsed command line parameters and send a startup success message to the main control module. The performance stress testing module is used to receive the inference command, construct multiple parameter combinations based on the parsed configuration parameters, perform inference tests based on each parameter combination, and send the inference results corresponding to each parameter combination to the main control module; wherein, the multiple parameter combinations include all combinations of the configuration parameters; The main control module is also used to receive the inference results corresponding to each parameter combination and send the inference results corresponding to each parameter combination to the output module. The output module is used to output the reasoning results corresponding to each parameter combination.

2. In the model testing system based on a multi-chip computing device as described in claim 1, before sending the inference results corresponding to each parameter combination to the output module, the main control module is further configured to process the inference results corresponding to each parameter combination in structured CSV format.

3. The model testing system based on a multi-chip computing device as described in claim 1, wherein the system includes an environment testing module, and the main control module is further configured to send an environment testing command to the environment testing module; The environment testing module is used to detect the hardware connection status, driver status, communication link status of the multi-chip computing device, and the installation status of the target inference framework on the multi-chip computing device.

4. The model testing system based on a multi-chip computing device as described in claim 3, wherein the environment testing module comprises: The driver detection unit is used to detect or install the required hardware drivers. A communication link detection unit is used to detect the communication status of RDMA or DMA in the multi-chip computing device; The computing unit detection unit is used to verify whether the matrix computing function of the multi-chip computing device is normal. The inference framework relies on a detection unit to verify the integrity and compatibility of the target inference framework installation.

5. The model testing system based on a multi-chip computing device as described in claim 1, wherein the test results corresponding to each parameter combination include: Request throughput, token throughput, first token latency, average latency of remaining tokens, and total inference time.

6. The model testing system based on a multi-chip computing device as described in claim 1, wherein the main control module is further configured to summarize the inference results corresponding to each parameter combination into a single file.

7. A model testing method based on a multi-chip computing device, applied to the model testing system based on a multi-chip computing device as described in any one of claims 1-6, the method comprising: The main control module obtains command line parameters and configuration parameters from the input module. The command line parameters represent the target chip number, target port, path of the model to be tested, and path of the target inference framework. The configuration parameters include the input token length, output token length, number of concurrent connections and samples, and number of repeated tests. The main control module parses the command-line parameters and the configuration parameters. Based on the main control module and the parsed command line parameters, a startup command is sent to the API service startup module; In response to receiving the startup success message from the API service startup module, an inference command is sent to the performance stress test module based on the main control module and the parsed configuration parameters. The system receives the inference results corresponding to each parameter combination sent by the performance stress testing module, and sends the inference results corresponding to each parameter combination to the output module. Each parameter combination is obtained based on the configuration parameters parsed by the performance stress testing module. The inference result corresponding to any parameter combination is obtained by the performance stress testing module using that parameter combination as the test condition.

8. The model testing method based on a multi-chip computing device as described in claim 7, wherein, Before sending the inference results corresponding to each parameter combination to the output module, the method further includes: Based on the main control module, the inference results corresponding to each parameter combination are processed into structured CSV format.

9. The model testing method based on a multi-chip computing device as described in claim 7, further comprising: In response to receiving an environment test command from the main control module, the system detects the hardware connection status, driver status, communication link status of the multi-chip computing device, and the installation status of the target inference framework on the multi-chip computing device based on the environment test module.

10. The model testing method based on a multi-chip computing device as described in claim 9, wherein the step of detecting the hardware connection status, driver status, communication link status of the multi-chip computing device, and the installation status of the target inference framework on the multi-chip computing device based on the environment testing module includes: The required hardware drivers are detected or installed based on the driver detection unit. The communication status of RDMA or DMA in the multi-chip computing device is detected based on the communication link detection unit. The matrix calculation function of the multi-chip computing device is verified based on the computing unit detection unit to ensure that it is normal. The inference framework dependency detection unit verifies the integrity and compatibility of the target inference framework installation.

11. The model testing method based on a multi-chip computing device as described in claim 7, wherein, The test results for each parameter combination include: request throughput, token throughput, first token latency, average latency of remaining tokens, and total inference time.

12. The model testing method based on a multi-chip computing device as described in claim 7, wherein, The method further includes: summarizing the inference results corresponding to each parameter combination into a single file based on the output module.

13. A computer-readable storage medium comprising instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 7-12.

14. A computing device comprising a processor having a computer program stored thereon, the computer program executing to implement the method as described in any one of claims 7-12.

Images