Model performance test method, device, equipment and medium

Abstract

The invention relates to the technical field of security testing, and discloses a model performance testing method, device, equipment and medium, which are used for sending pressure data to a code task model, receiving GPU load, token processing efficiency and response time fed back by the code task model, and dynamically adjusting the pressure data to ensure reasonable utilization of hardware resources and explore limit performance of the model. When the GPU load is too high, the pressure is reduced, when the response time exceeds the standard, the user load capacity limit value is determined, when the token processing efficiency reaches the peak value, the maximum processing efficiency is determined, and finally the two key indexes serve as test results. According to the method, the pressure data is adjusted in a multi-gradient manner, so that the pressure data is closer to dynamic irregular fluctuation of concurrency in a real production environment, and the accuracy of a model test result is improved.

Description

Technical Field

[0001] This invention relates to the field of security testing technology, and in particular to model performance testing methods, apparatus, equipment and media. Background Technology

[0002] With the large-scale application of code task models in code task scenarios such as code completion, intelligent question answering, and automated programming, VLLM-based inference services have become the core technical architecture supporting high-concurrency code task requests.

[0003] Traditional performance testing methods for code task models typically employ a fixed concurrent pressure gradient for stress testing. However, the concurrency in real production environments fluctuates dynamically and irregularly, resulting in test results that fail to accurately reflect the production environment's carrying capacity.

[0004] Therefore, in the face of the increasing demand for performance testing of code task models, the current performance testing methods for code task models urgently need to be improved to solve the problem of insufficient accuracy of test results from existing methods. Summary of the Invention

[0005] This invention provides a model performance testing method, apparatus, equipment, and medium, which mainly solves the problem of insufficient accuracy of model performance testing results.

[0006] Firstly, a model performance testing method is provided, including: Acquire stress data for testing the load of the code task model, and send the stress data to the preset code task model; Receive feedback from the code task model regarding GPU load, token processing efficiency, and response time under the stress data; When the GPU load exceeds a preset range, the stress data is adjusted by a factor to obtain adjusted stress data, and the process returns to the step of sending the stress data to the preset code task model; when the GPU load is within a preset range and the response time exceeds a preset response threshold, the stress data corresponding to the response time is used as the user load capacity limit value. The adjusted pressure data is adjusted again by a factor to obtain readjusted pressure data, and the process returns to the step of sending the pressure data to the preset code task model. When the GPU load is within a preset range, the response time exceeds a preset response threshold, and the tokens processing efficiency reaches a preset peak value, the peak value is taken as the maximum tokens processing efficiency of the code task model under preset conditions. The user load capacity limit and the maximum tokens processing efficiency are used as the performance test results.

[0007] Secondly, a model performance testing device is provided, comprising: The stress data sending module is used to acquire stress data for testing the load of the code task model and send the stress data to the preset code task model. The receiving module is used to receive the GPU load, token processing efficiency, and response time under the stress data fed back by the code task model; The stress data multiplier adjustment module is used to adjust the stress data by a multiplier when the GPU load exceeds a preset range, obtain adjusted stress data, and return to the step of sending the stress data to a preset code task model. The load capacity limit value acquisition module is used to take the pressure data corresponding to the response time as the user load capacity limit value when the GPU load is within a preset range and the response time exceeds a preset response threshold. The pressure data multiplier readjustment module is used to readjust the adjusted pressure data by a multiplier again to obtain readjusted pressure data, and then return to the step of sending the pressure data to the preset code task model. The maximum tokens processing efficiency acquisition module is used to take the peak value as the maximum tokens processing efficiency of the code task model under preset conditions when the GPU load is within a preset range, the response time exceeds a preset response threshold, and the tokens processing efficiency reaches a preset peak value. The performance test result acquisition module is used to take the user load capacity limit value and the maximum tokens processing efficiency as the performance test result.

[0008] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the aforementioned model performance testing method.

[0009] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the aforementioned model performance testing method.

[0010] In the aforementioned model performance testing method, apparatus, computer equipment, and storage medium, by acquiring stress data and sending it to a preset code task model, the standardization and repeatability of the test are ensured, while facilitating systematic analysis of model performance. When the GPU load exceeds a preset range, the stress data is adjusted by a factor, avoiding blindly increasing or decreasing the stress and helping to more accurately find the model's performance critical point, thereby improving the accuracy of the test results. The adjusted stress data is then adjusted by a factor again to obtain readjusted stress data. After determining the initial safe load, the stress data is adjusted by a factor again to explore the model's performance when approaching its limit, fully exploring the model's hardware resource potential and further improving the accuracy of the model test results. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of an application environment for a model performance testing method according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating a model performance testing method according to one embodiment of the present invention; Figure 3 This is a schematic diagram of a model performance testing device according to one embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a computer device according to one embodiment of the present invention; Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention.

[0013] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0014] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0015] The model performance testing method provided in this embodiment of the invention can be applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network. The server can dynamically adjust stress data, monitor GPU load, token processing efficiency, and response time in real time to determine the user load capacity limit, determine the maximum processing efficiency, and feed back the target model performance test results to the client. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0016] Please see Figure 2 As shown, Figure 2 A flowchart illustrating a model performance testing method provided in an embodiment of the present invention includes the following steps: S1. Obtain the stress data used to test the load of the code task model, and send the stress data to the preset code task model.

[0017] In this embodiment of the invention, the stress data refers to all input data used to load test the code task model, such as 10, 20, 30; the code task model refers to an artificial intelligence model specifically optimized for code-related tasks.

[0018] In this embodiment of the invention, a code task model load of different magnitudes is generated by a stress testing tool (such as JMeter) to simulate concurrent requests or continuous high load scenarios; then, the generated stress data (such as request volume, task processing time, error rate, etc.) is sent to the preset code task model through an API interface or data transmission protocol to evaluate its performance and stability under different load conditions.

[0019] In this embodiment of the invention, sending the stress data to a preset code task model includes: Determine the cooling interval for transmitting pressure data; The pressure gradient data is sent to each of the preset code task models according to the cooling interval.

[0020] In this embodiment of the invention, the parameters of the pressure data are first determined, including the initial value, tolerance and the number of data, and multiple pressure gradient data in the form of an arithmetic sequence are generated. Then, the cooling interval for sending the pressure data is set, and each pressure gradient data is sent to a preset code task model in sequence according to the interval to simulate performance testing under different load conditions.

[0021] In this embodiment of the invention, the pressure gradient of an arithmetic sequence can simulate the linear growth process of system load from low to high, which is closer to the actual user scenario than suddenly sending high loads. It can more accurately test the system's load capacity and response changes. The pressure gradient data changes in a regular manner, which makes it easier for testers to analyze the system's performance under different loads, such as the changing trends of indicators like CPU utilization, memory usage, and response time. The setting of the cooling interval can control the frequency of pressure data transmission, avoiding excessive impact on the target system and ensuring the stability and repeatability of the test process. By sending data at intervals, processing time can be given to the target system, reducing resource contention and inaccurate test results caused by continuously sending a large number of requests.

[0022] S2. Receive feedback from the code task model on GPU load, token processing efficiency, and response time under the pressure data.

[0023] In this embodiment of the invention, the GPU load data, token processing efficiency indicators, and response time information fed back in real time by the code task model during the processing of sent stress data are first continuously received through a preset communication interface or data receiving module. Then, these feedback data are collected and organized to evaluate the performance of the model under different stress conditions.

[0024] In this embodiment of the invention, after receiving the GPU load, token processing efficiency, and response time under the stress data fed back by the code task model, the method further includes: Receive the network transmission volume under the pressure data as fed back by the code task model; Obtain the sending timestamp of the pressure data and the receiving timestamp of the pressure data, and calculate the time difference between the sending timestamp and the receiving timestamp; Calculate the ratio of the network transmission volume to the time difference, and use the ratio as the network throughput of xx.

[0025] In this embodiment of the invention, the network transmission volume usually refers to the total amount of data transmitted between the model and external systems (such as data centers and cloud services), and the unit may be bytes or kilobytes (KB).

[0026] In this embodiment of the invention, during the process of receiving network transmission volume, the sending end of the network communication (e.g., the client that initiates the stress test) first needs to record the total amount of data successfully sent within a specific time interval (e.g., 1 second, 10 seconds). The total amount of data can be obtained from the network statistics information of the operating system (e.g., ifstat), the application's log records, or a dedicated network monitoring tool.

[0027] Furthermore, the precise time point at which the sending end begins sending the data block (or the entire time window of data) is recorded, typically in the format YYYY-MM-DDHH:MM:SS.FFFFFF (inclusive of microseconds). The precise time point at which the receiving end confirms successful receipt of the data block (or the entire time window of data) is also recorded. The time interval between the two events is obtained by subtracting the sending timestamp from the receiving timestamp.

[0028] Furthermore, the network transmission volume (bytes) is divided by the time difference (seconds) to obtain the network transmission volume.

[0029] For example, after the sending end starts sending data and continues for 10 seconds, the statistics at the receiving end are checked after the sending ends, and it is found that a total of 81,920,000 bytes of data were successfully received, that is, the network transmission volume was 81,920,000 bytes. Furthermore, the timestamp of the sending end starting to send is: 2023-10-27 15:30:00.000000, and the timestamp of the receiving end confirming the completion of receiving is: 2023-10-27 15:30:10.000000, with a time difference of 10.000000 seconds; Furthermore, the throughput (Byte / s) was calculated as 81,920,000 Bytes / 10,000,000 s = 8,192,000 Bytes / s. Then, a unit conversion was performed to more commonly used units, resulting in MB / s: 8,192,000 Bytes / s / 1,048,576 Bytes / MB ≈ 7.8125 MB / s. In this test, the throughput of this network interface was approximately 7.8125 MB / s, or 65.536 Mbps.

[0030] In this embodiment of the invention, by specifying the total amount of data within a time window, the "quantity" of network transmission can be quantified; by using high-precision timestamps to calculate the time difference of data transmission, the transmission time can be accurately reflected; by dividing the two, the throughput index can be obtained to comprehensively evaluate the actual transmission speed and efficiency of the network within a specific time period, providing a reliable basis for network performance optimization and capacity planning.

[0031] In this embodiment of the invention, after receiving the GPU load, token processing efficiency, and response time under the stress data fed back by the code task model, the method further includes: Determine the load threshold for the GPU load; Identify abnormal GPU loads exceeding the specified load threshold within the GPU load; Identify the stress data corresponding to the abnormal GPU load, and determine the model critical point based on the corresponding stress data.

[0032] In this embodiment of the invention, in setting the GPU load threshold, firstly, a reasonable GPU load baseline value needs to be determined based on the model hardware environment (such as GPU model and server configuration) and the expected usage scenario. Based on the baseline value, an upper limit is set as the load threshold. This threshold represents the maximum GPU resource utilization rate that the model can stably and efficiently run on this hardware.

[0033] Furthermore, the load threshold types include absolute thresholds and relative thresholds. An absolute threshold can be a fixed percentage, such as 80%; when the GPU load exceeds this value, the system is considered to be facing a performance bottleneck. A relative threshold can be a percentage based on a baseline value, such as 100%, indicating that when the GPU load reaches 1.1 times its historical average level, it begins to enter an unstable region.

[0034] Furthermore, setting the threshold requires comprehensive consideration of multiple factors, including: hardware characteristics, business requirements, and performance metrics. Hardware characteristics: different GPUs have different heat dissipation capabilities and parallel computing efficiencies. Business requirements: for scenarios with high real-time requirements (such as code completion), the threshold should be set lower to ensure response time. For scenarios with low non-real-time requirements (such as batch processing), the threshold can be set higher to improve resource utilization. Performance metrics: refer to historical data or preliminary tests to ensure that the threshold reflects performance bottlenecks without being too conservative or too aggressive.

[0035] Furthermore, in the process of identifying abnormal GPU loads exceeding the load threshold, the real-time load data of each GPU card is first obtained from the code task model monitoring system in real time or periodically during the stress test. The real-time load data of each GPU is compared with the set load threshold one by one. If the real-time load value of a certain GPU is greater than the load threshold, the load of that GPU is determined to be abnormal. If the load of all GPUs is lower than the threshold, the current stress level is considered to be within an acceptable range and there is no abnormal load.

[0036] Furthermore, for the identified abnormal GPU load, the timestamp of the abnormal GPU load, GPUID (which GPU card is abnormal), abnormal load value (the actual load percentage of the GPU), and corresponding stress data are recorded, including the number of concurrent users when the abnormality occurred, request type (code completion / Q&A), etc.

[0037] In this embodiment of the invention, the model critical point refers to the pressure level at which the model performance begins to decline significantly (such as reduced throughput or prolonged response time) or system resources (such as GPU) begin to be overloaded. Based on the timestamp of the recorded abnormal GPU load and the corresponding pressure data, the specific pressure data point at the time of the abnormality is traced back in the test log.

[0038] Furthermore, in determining the model's critical point based on stress data, testing begins with lower stress data (stress gradients). GPU load and other performance metrics are recorded for each gradient. When an abnormal GPU load is detected at a certain gradient, the stress level of that gradient is used as the lower limit of the critical stress value. Then, near this critical stress value, finer stress gradients are set (e.g., adding gradients of 85, 90, 95, etc. between 80 and 100 concurrent users) for further testing to determine the precise stress point where performance begins to decline. Finally, the following are recorded: Concurrent Users: the maximum number of concurrent users at the critical point; Throughput: the average throughput at the critical point; Response Time: the average response time at the critical point; GPU Load: the average GPU load at the critical point.

[0039] In this embodiment of the invention, by setting a GPU load threshold, the performance boundary of model operation can be clearly defined, enabling precise monitoring of the system status; identifying abnormal GPU load can provide timely warnings of potential performance bottlenecks, ensuring system stability; and determining the model critical point based on abnormal load-related pressure data can accurately locate the critical pressure level of model performance degradation, providing a key basis for system capacity planning and performance optimization.

[0040] S3. Determine whether the GPU load exceeds a preset range.

[0041] In this embodiment of the invention, during the stress test, GPU load data of the code task model is collected in real time and compared with a preset range (including a lower threshold and an upper threshold) to determine whether the real-time GPU load exceeds the range.

[0042] S4. Adjust the pressure data by a factor to obtain adjusted pressure data, and return to S1.

[0043] In this embodiment of the invention, the preset interval can be a performance state interval division to define the current performance state of the model server (such as normal, overloaded or about to be overloaded). When the actual GPU load exceeds this interval range, the corresponding adjustment strategy is triggered. The interval is [lower threshold, upper threshold].

[0044] In this embodiment of the invention, adjusting the pressure data by a factor to obtain adjusted pressure data includes: Determine whether the GPU load exceeds the upper limit of a preset range; When the GPU load exceeds the upper limit of the range, the pressure data is reduced by a preset first ratio, and the reduced pressure data is used as the adjustment pressure data. When the GPU load does not exceed the upper limit of the range, the pressure data is amplified by a preset second ratio, and the amplified pressure data is used as the adjustment pressure data.

[0045] In this embodiment of the invention, a pre-set upper limit for the GPU load range (e.g., 85%) is first set. During the stress test, the GPU load value (e.g., real-time load 90%) under the current stress data (e.g., 50 concurrent users) is collected in real time. The real-time GPU load value is compared with the preset upper limit of the range to obtain a judgment result of "exceeding" or "not exceeding". This result directly determines the direction of subsequent stress adjustment.

[0046] Furthermore, when the GPU load exceeds the upper limit of the interval, the pressure data is reduced by a first proportional multiple. The current pressure data (such as concurrency and request frequency) is multiplied by the first proportional multiple to obtain the reduced adjusted pressure data. The reduced pressure data is used as the new test pressure to re-initiate requests to the code task model, while continuously monitoring the adjusted GPU load, token processing efficiency and other indicators.

[0047] Furthermore, when the GPU load does not exceed the upper limit of the range, the stress data is amplified by the second proportional coefficient. The current stress data is multiplied by the second proportional coefficient to obtain the amplified adjusted stress data. The amplified stress data is used as the new test stress, and the request is re-initiated to continue monitoring core indicators such as GPU load and response time.

[0048] In this embodiment of the invention, after adjusting the pressure data by a factor to obtain adjusted pressure data, the method further includes: The GPU load, the tokens processing efficiency, and the response time are normalized respectively to obtain the normalized load corresponding to the GPU load, the normalized tokens efficiency corresponding to the tokens processing efficiency, and the normalized response time corresponding to the response time. The normalized load, the normalized tokens efficiency, and the normalized response time are then concatenated into a multi-dimensional performance index vector. Calculate the deviation vector between the multidimensional performance index vector and the preset index vector; The pressure data is adjusted according to the preset pressure index mapping library and the deviation vector to obtain adjusted pressure data.

[0049] In this embodiment of the invention, during the normalization process of GPU load, token processing efficiency, and response time, the performance indicators of GPU load, token processing efficiency, and response time are converted into uniform values ​​in the [0,1] range to eliminate the magnitude difference between indicators, allowing multi-dimensional indicators to be directly compared and calculated. Through historical testing or pre-testing, the minimum and maximum values ​​of each indicator (i.e., the reasonable fluctuation boundaries of the indicator) are determined. The minimum and maximum values ​​of each indicator are substituted into the normalization formula to obtain the normalization results of each indicator.

[0050] Furthermore, by pre-defining the order of vector dimensions (e.g., [normalized load, normalized tokens efficiency, normalized response time]), the three normalized values ​​are combined sequentially to form a vector of the form... A multidimensional performance index vector, ( =Normalized load, =Normalized tokens efficiency =Normalized response time).

[0051] Furthermore, in the process of calculating the deviation vector, such as in a code completion scenario, the preset metric vector is: moderate GPU load (e.g., 80%, normalized = 0.8), highest token efficiency (50 tokens / s, normalized = 1), and shortest response time (200ms, normalized = 1). The preset vector is... ; Deviation vector = Performance index vector - Preset index vector, that is Each dimension of the deviation vector represents the deviation between the current metric and the ideal metric: a positive value indicates "exceeding the ideal value" (e.g., load normalization value 0.9 - ideal 0.8 = 0.1, indicating that the load is too high), and a negative value indicates "not reaching the ideal value" (e.g., tokens efficiency normalization value 0.7 - ideal 1 = -0.3, indicating that the efficiency is too low).

[0052] Furthermore, based on the deviation vector and referring to the preset mapping rules, the stress data is precisely adjusted. First, a stress indicator mapping library is constructed: the mapping library is a pre-established correspondence of "deviation vector feature → stress adjustment rule". The core is to define the proportion / direction of stress adjustment according to the deviation direction / degree of each dimension. For example, deviation vector feature (core dimension): positive deviation of load dimension (deviation value > 0, load is too high), the corresponding stress adjustment rule: stress data is reduced by 0.8 times; deviation vector feature (core dimension): negative deviation of tokens efficiency (deviation value < -0.2, efficiency is significantly low), the corresponding stress adjustment rule: stress data is reduced by 0.7 times.

[0053] Furthermore, the corresponding adjustment rules are matched from the mapping library to calculate the new adjustment pressure data (e.g., if the original pressure data is 80 concurrent users, the "high load" rule is matched to adjust it to 80 × 0.8 = 64 concurrent users).

[0054] In this embodiment of the invention, performance metrics with different numerical ranges (such as GPU load, token processing efficiency, and response time) are converted into a unified [0,1] interval value, eliminating the magnitude differences between metrics and enabling direct comparison and comprehensive evaluation of multi-dimensional metrics. By comparing the current multi-dimensional performance metric vector with a preset ideal metric vector, the gap between the current performance state and the "optimal state" is quantified, clarifying which dimensions the model performs poorly in (such as excessive load, low efficiency, and slow response), providing a clear direction and basis for subsequent stress adjustments. Based on the performance shortcomings reflected by the deviation vector and referring to a preset stress metric mapping library, refined and intelligent adjustments to stress data are achieved. This adjustment is not a simple scaling up or down, but rather a targeted optimization of the stress strategy based on the model's comprehensive performance across multiple dimensions, thereby more effectively exploring the model's performance boundaries and finding the optimal concurrent capacity.

[0055] S5. Determine whether the response time exceeds a preset response threshold.

[0056] In this embodiment of the invention, during stress testing, the response time of the code task model under specific stress data is collected in real time and compared with a preset response threshold to determine whether the real-time response time exceeds the threshold range.

[0057] S6. Use the pressure data corresponding to the response time as the user load capacity limit value, adjust the adjustment pressure data again by a factor to obtain the readjustment pressure data, and return to S1.

[0058] In this embodiment of the invention, the response time includes streaming request response time and non-streaming request response time, and the preset response threshold includes streaming request response threshold and non-streaming request response threshold. When the GPU load is within a preset range and the response time exceeds the preset response threshold, the stress data corresponding to the response time is used as the user load capacity limit value. The method then includes: When the streaming request response time exceeds the streaming request response threshold, the pressure data corresponding to the streaming request response time is used as the user load capacity limit value under the streaming request. When the non-streaming request response time exceeds the non-streaming request response threshold, the pressure data corresponding to the non-streaming request response time is used as the user load capacity limit value under non-streaming requests.

[0059] In this embodiment of the invention, the streaming request can be a question-and-answer dialogue scenario, where the model returns the result word by word or sentence by sentence, and the user can see the output in real time. The non-streaming request can be a code completion scenario, where the model first generates a complete code snippet and then returns it to the user all at once.

[0060] Furthermore, for each request type, based on business needs and user experience standards, set a maximum acceptable response time as a threshold. For non-streaming requests, the threshold is: for code completion, users typically expect to see results within a few hundred milliseconds. Based on your description, a strict threshold can be set, such as 500ms. This means that a complete code completion request, from the user sending the request to the model returning all results, should be completed within 500ms in 95% of cases (i.e., the vast majority of requests). If this time exceeds, the user experience will significantly degrade.

[0061] Furthermore, regarding streaming request thresholds: For streaming Q&A, users are very sensitive to the waiting time for the "first byte"; it is generally considered that a good user experience is achieved when the first byte is returned within 300ms. Therefore, the first byte response time threshold for streaming requests can be set to 300ms.

[0062] Furthermore, during stress testing, the first byte response time of streaming requests is monitored in real time and compared with a preset streaming request threshold to determine whether the performance of this request type meets the standard under the current pressure. When the first byte response time of a streaming request reaches the threshold, the stress data at this time (such as the number of concurrent users) is recorded. This data is the maximum safe user load capacity limit value under the streaming request scenario. In this embodiment of the invention, the steps of using the pressure data corresponding to the non-streaming request response time as the user load capacity limit value under the non-streaming request are the same as those of using the pressure data corresponding to the streaming request response time as the user load capacity limit value under the streaming request, and will not be described in detail here.

[0063] In this embodiment of the invention, during the stress data test, if the GPU load does not exceed the preset range and the response time exceeds the preset threshold, the current adjustment stress data is amplified again by a preset third proportional coefficient (greater than 1) to obtain readjustment stress data. This readjustment stress data is then sent back to the code task model as a new test stress. At the same time, the GPU load, tokens processing efficiency, and response time under the readjustment stress data are continuously monitored and received from the model until the tokens processing efficiency reaches the preset peak value, that is, the tokens processing efficiency changes from increasing to decreasing.

[0064] S7. Determine whether the processing efficiency of the tokens has reached the preset peak value.

[0065] In this embodiment of the invention, during the stress test, the peak value of the code task model under specific stress data is collected in real time and compared with the preset peak value to determine whether the peak value exceeds the threshold range.

[0066] S8. Continuously adjust the readjustment pressure data to obtain optimized adjustment pressure data, and return to S1.

[0067] In this embodiment of the invention, the step of continuously adjusting the readjustment pressure data to obtain optimized adjustment pressure data is the same as the step of adjusting the adjustment pressure data by a factor of two again to obtain readjustment pressure data, and will not be described in detail here.

[0068] S9. The peak value is taken as the maximum tokens processing efficiency of the code task model under preset conditions.

[0069] In this embodiment of the invention, during the stress test, the optimized stress data is continuously sent to the code task model and the GPU load, token processing efficiency and response time are received from it. When it is detected that the GPU load does not exceed the preset range, the response time exceeds the preset response threshold and the token processing efficiency reaches the preset peak, the test is stopped immediately and the peak value is taken as the maximum token processing efficiency of the code task model under the preset conditions.

[0070] S10. The user load capacity limit and the maximum tokens processing efficiency are used as the performance test results.

[0071] In this embodiment of the invention, the performance test results include two key indicators: one is the user load capacity limit value, which represents the maximum number of concurrent users that the model can stably handle without violating the preset response time threshold; the other is the maximum tokens processing efficiency, which represents the maximum number of tokens that the model can process per unit time. These two indicators together reflect the performance boundaries and processing capabilities of the model under the current test conditions, providing a comprehensive and valuable reference for the actual deployment and performance optimization of the model.

[0072] As can be seen, in the above scheme, for the business of building model performance test results, pressure data for testing the load of code task model is obtained and the pressure data is sent to the preset code task model; the GPU load, token processing efficiency and response time under the pressure data are received from the code task model; when the GPU load exceeds the preset range, the pressure data is adjusted by a factor to obtain adjusted pressure data, and the process of sending the pressure data to the preset code task model is returned. When the GPU load is within a preset range and the response time exceeds a preset response threshold, the stress data corresponding to the response time is used as the user load capacity limit value; the adjusted stress data is adjusted again by a factor to obtain re-adjusted stress data, and the step of sending the stress data to the preset code task model is returned; when the GPU load is within a preset range, the response time exceeds the preset response threshold, and the tokens processing efficiency reaches a preset peak value, the peak value is used as the maximum tokens processing efficiency of the code task model under preset conditions; the user load capacity limit value and the maximum tokens processing efficiency are used as performance test results. By adjusting the stress data through multiple gradients, it is made to more closely resemble the dynamic and irregular fluctuations of concurrency in a real production environment, thereby improving the accuracy of the model test results.

[0073] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0074] In one embodiment, a model performance testing apparatus is provided, which corresponds one-to-one with a model performance testing method described in the above embodiments. For example... Figure 3 As shown, this model performance testing device includes a pressure data transmission module 101, a receiving module 102, a pressure data multiplier adjustment module 103, a load capacity limit value acquisition module 104, a pressure data multiplier readjustment module 105, a maximum tokens processing efficiency acquisition module 106, and a performance test result acquisition module 107. Detailed descriptions of each functional module are as follows: The stress data sending module 101 is used to acquire stress data for testing the load of the code task model and send the stress data to a preset code task model. The receiving module 102 is used to receive the GPU load, token processing efficiency and response time under the stress data fed back by the code task model; The pressure data multiplier adjustment module 103 is used to adjust the pressure data by a multiplier when the GPU load exceeds a preset range, obtain adjusted pressure data, and return to the step of sending the pressure data to a preset code task model. The load capacity limit value acquisition module 104 is used to take the pressure data corresponding to the response time as the user load capacity limit value when the GPU load is in a preset range and the response time exceeds a preset response threshold. The pressure data multiplier readjustment module 105 is used to readjust the adjusted pressure data again to obtain readjusted pressure data, and return to the step of sending the pressure data to the preset code task model. The maximum tokens processing efficiency acquisition module 106 is used to take the peak value as the maximum tokens processing efficiency of the code task model under preset conditions when the GPU load is in a preset range, the response time exceeds a preset response threshold, and the tokens processing efficiency reaches a preset peak value. The performance test result acquisition module 107 is used to take the user load capacity limit value and the maximum tokens processing efficiency as the performance test result.

[0075] In one embodiment, the stress data sending module 101, when sending the stress data to a preset code task model, is used to: Determine the cooling interval for transmitting pressure data; The pressure gradient data is sent to each of the preset code task models according to the cooling interval.

[0076] In one embodiment, the receiving module 102, when performing the task of receiving feedback from the code task model regarding GPU load, token processing efficiency, and response time under the stress data, is configured to: Determine the load threshold for the GPU load; Identify abnormal GPU loads exceeding the specified load threshold within the GPU load; Identify the stress data corresponding to the abnormal GPU load, and determine the model critical point based on the corresponding stress data.

[0077] In one embodiment, the receiving module 102, when performing the task of receiving feedback from the code task model regarding GPU load, token processing efficiency, and response time under the stress data, is configured to: Receive the network transmission volume under the pressure data as fed back by the code task model; Obtain the sending timestamp of the pressure data and the receiving timestamp of the pressure data, and calculate the time difference between the sending timestamp and the receiving timestamp; Calculate the ratio of the network transmission volume to the time difference, and use the ratio as the network throughput under the pressure data.

[0078] In one embodiment, when the pressure data multiplier adjustment module 103 adjusts the pressure data to obtain adjusted pressure data, it is used to: Determine whether the GPU load exceeds the upper limit of a preset range; When the GPU load exceeds the upper limit of the range, the pressure data is reduced by a preset first ratio, and the reduced pressure data is used as the adjustment pressure data. When the GPU load does not exceed the upper limit of the range, the pressure data is amplified by a preset second ratio, and the amplified pressure data is used as the adjustment pressure data.

[0079] In one embodiment, when the pressure data multiplier adjustment module 103 performs a multiplier adjustment on the pressure data to obtain adjusted pressure data, it is used to: The GPU load, the tokens processing efficiency, and the response time are normalized respectively to obtain the normalized load corresponding to the GPU load, the normalized tokens efficiency corresponding to the tokens processing efficiency, and the normalized response time corresponding to the response time. The normalized load, the normalized tokens efficiency, and the normalized response time are then concatenated into a multi-dimensional performance index vector. Calculate the deviation vector between the multidimensional performance index vector and the preset index vector; The pressure data is adjusted according to the preset pressure index mapping library and the deviation vector to obtain adjusted pressure data.

[0080] In one embodiment, the load capacity limit acquisition module 104, when executing the response time (including streaming request response time and non-streaming request response time) and the preset response threshold (including streaming request response threshold and non-streaming request response threshold), uses the stress data corresponding to the response time as the user load capacity limit value when the GPU load is within a preset range and the response time exceeds the preset response threshold, for the following purposes: When the streaming request response time exceeds the streaming request response threshold, the pressure data corresponding to the streaming request response time is used as the user load capacity limit value under the streaming request. When the non-streaming request response time exceeds the non-streaming request response threshold, the pressure data corresponding to the non-streaming request response time is used as the user load capacity limit value under non-streaming requests.

[0081] This invention provides a model performance testing device. For the business of building model performance test results, it acquires stress data for testing the load of code task models and sends the stress data to a preset code task model; it receives feedback from the code task model on GPU load, token processing efficiency, and response time under the stress data; when the GPU load exceeds a preset range, it adjusts the stress data by a factor to obtain adjusted stress data, and returns to the step of sending the stress data to the preset code task model. When the GPU load is within a preset range and the response time exceeds a preset response threshold, the stress data corresponding to the response time is used as the user load capacity limit value; the adjusted stress data is adjusted again by a factor to obtain re-adjusted stress data, and the step of sending the stress data to the preset code task model is returned; when the GPU load is within a preset range, the response time exceeds the preset response threshold, and the tokens processing efficiency reaches a preset peak value, the peak value is used as the maximum tokens processing efficiency of the code task model under preset conditions; the user load capacity limit value and the maximum tokens processing efficiency are used as performance test results. By adjusting the stress data through multiple gradients, it is made to more closely resemble the dynamic and irregular fluctuations of concurrency in a real production environment, thereby improving the accuracy of the model test results.

[0082] For specific limitations regarding a model performance testing device, please refer to the limitations regarding a model performance testing method described above, which will not be repeated here. Each module in the aforementioned model performance testing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0083] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a model performance testing method on the server side.

[0084] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a model performance testing method on the client side.

[0085] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Acquire stress data for testing the load of the code task model, and send the stress data to the preset code task model; Receive feedback from the code task model regarding GPU load, token processing efficiency, and response time under the stress data; When the GPU load exceeds a preset range, the stress data is adjusted by a factor to obtain adjusted stress data, and the process returns to the step of sending the stress data to the preset code task model; when the GPU load is within a preset range and the response time exceeds a preset response threshold, the stress data corresponding to the response time is used as the user load capacity limit value. The adjusted pressure data is adjusted again by a factor to obtain readjusted pressure data, and the process returns to the step of sending the pressure data to the preset code task model. When the GPU load is within a preset range, the response time exceeds a preset response threshold, and the tokens processing efficiency reaches a preset peak value, the peak value is taken as the maximum tokens processing efficiency of the code task model under preset conditions. The user load capacity limit and the maximum tokens processing efficiency are used as the performance test results.

[0086] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Acquire stress data for testing the load of the code task model, and send the stress data to the preset code task model; Receive feedback from the code task model regarding GPU load, token processing efficiency, and response time under the stress data; When the GPU load exceeds a preset range, the stress data is adjusted by a factor to obtain adjusted stress data, and the process returns to the step of sending the stress data to the preset code task model; when the GPU load is within a preset range and the response time exceeds a preset response threshold, the stress data corresponding to the response time is used as the user load capacity limit value. The adjusted pressure data is adjusted again by a factor to obtain readjusted pressure data, and the process returns to the step of sending the pressure data to the preset code task model. When the GPU load is within a preset range, the response time exceeds a preset response threshold, and the tokens processing efficiency reaches a preset peak value, the peak value is taken as the maximum tokens processing efficiency of the code task model under preset conditions. The user load capacity limit and the maximum tokens processing efficiency are used as the performance test results.

[0087] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0088] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0089] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0090] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. If any software tools or components other than those of our company appear in the embodiments, they are merely illustrative examples and do not represent actual use. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A model performance testing method, characterized in that, include: Acquire stress data for testing the load of the code task model, and send the stress data to the preset code task model; Receive feedback from the code task model regarding GPU load, token processing efficiency, and response time under the stress data; When the GPU load exceeds a preset range, the stress data is adjusted by a factor to obtain adjusted stress data, and the process returns to the step of sending the stress data to the preset code task model; when the GPU load is within a preset range and the response time exceeds a preset response threshold, the stress data corresponding to the response time is used as the user load capacity limit value. The adjusted pressure data is adjusted again by a factor to obtain readjusted pressure data, and the process returns to the step of sending the pressure data to the preset code task model. When the GPU load is within a preset range, the response time exceeds a preset response threshold, and the tokens processing efficiency reaches a preset peak value, the peak value is taken as the maximum tokens processing efficiency of the code task model under preset conditions. The user load capacity limit and the maximum tokens processing efficiency are used as the performance test results.

2. The model performance testing method as described in claim 1, characterized in that, After receiving feedback from the code task model regarding GPU load, token processing efficiency, and response time under the stress data, the method further includes: Receive the network transmission volume under the pressure data as fed back by the code task model; Obtain the sending timestamp of the pressure data and the receiving timestamp of the pressure data, and calculate the time difference between the sending timestamp and the receiving timestamp; Calculate the ratio of the network transmission volume to the time difference, and use the ratio as the network throughput under the pressure data.

3. The model performance testing method as described in claim 1, characterized in that, The step of adjusting the pressure data by a factor to obtain adjusted pressure data includes: Determine whether the GPU load exceeds the upper limit of a preset range; When the GPU load exceeds the upper limit of the range, the pressure data is reduced by a preset first ratio, and the reduced pressure data is used as the adjustment pressure data. When the GPU load does not exceed the upper limit of the range, the pressure data is amplified by a preset second ratio, and the amplified pressure data is used as the adjustment pressure data.

4. The model performance testing method as described in claim 1, characterized in that, After adjusting the pressure data by a factor to obtain adjusted pressure data, the method further includes: The GPU load, the tokens processing efficiency, and the response time are normalized respectively to obtain the normalized load corresponding to the GPU load, the normalized tokens efficiency corresponding to the tokens processing efficiency, and the normalized response time corresponding to the response time. The normalized load, the normalized tokens efficiency, and the normalized response time are then concatenated into a multi-dimensional performance index vector. Calculate the deviation vector between the multidimensional performance index vector and the preset index vector; The pressure data is adjusted according to the preset pressure index mapping library and the deviation vector to obtain adjusted pressure data.

5. The model performance testing method as described in claim 1, characterized in that, The response time includes streaming request response time and non-streaming request response time. The preset response threshold includes streaming request response threshold and non-streaming request response threshold. When the GPU load is within a preset range and the response time exceeds the preset response threshold, the stress data corresponding to the response time is used as the user load capacity limit value. The method includes: When the streaming request response time exceeds the streaming request response threshold, the pressure data corresponding to the streaming request response time is used as the user load capacity limit value under the streaming request. When the non-streaming request response time exceeds the non-streaming request response threshold, the pressure data corresponding to the non-streaming request response time is used as the user load capacity limit value under non-streaming requests.

6. The model performance testing method as described in claim 1, characterized in that, After receiving feedback from the code task model regarding GPU load, token processing efficiency, and response time under the stress data, the method further includes: Determine the load threshold for the GPU load; Identify abnormal GPU loads exceeding the specified load threshold within the GPU load; Identify the stress data corresponding to the abnormal GPU load, and determine the model critical point based on the corresponding stress data.

7. The model performance testing method as described in claim 1, characterized in that, Sending the stress data to the preset code task model includes: Determine the cooling interval for transmitting pressure data; The pressure gradient data is sent to each of the preset code task models according to the cooling interval.

8. A model performance testing device, characterized in that, include: The stress data sending module is used to acquire stress data for testing the load of the code task model and send the stress data to the preset code task model. The receiving module is used to receive the GPU load, token processing efficiency, and response time under the stress data fed back by the code task model; The stress data multiplier adjustment module is used to adjust the stress data by a multiplier when the GPU load exceeds a preset range, obtain adjusted stress data, and return to the step of sending the stress data to a preset code task model. The load capacity limit value acquisition module is used to take the pressure data corresponding to the response time as the user load capacity limit value when the GPU load is within a preset range and the response time exceeds a preset response threshold. The pressure data multiplier readjustment module is used to readjust the adjusted pressure data again to obtain readjusted pressure data, and return to the step of sending the pressure data to the preset code task model; The maximum tokens processing efficiency acquisition module is used to take the peak value as the maximum tokens processing efficiency of the code task model under preset conditions when the GPU load is within a preset range, the response time exceeds a preset response threshold, and the tokens processing efficiency reaches a preset peak value. The performance test result acquisition module is used to take the user load capacity limit value and the maximum tokens processing efficiency as the performance test result.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the model performance testing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the model performance testing method as described in any one of claims 1 to 7.

Images