Configuration method, system and device, program product and storage medium

Abstract

The invention discloses a configuration method, system and device, a program product and a storage medium. The method comprises the steps of obtaining a large language model LLM type and demand information; generating configuration parameters based on the LLM type and the demand information; creating a test service based on the configuration parameters; performing a pressure test on the test service by using a test script to obtain a pressure test result; and adjusting the configuration parameters based on the pressure measurement result to obtain the configuration parameters meeting preset performance requirements. The resource allocation problem caused by lack of professional knowledge is effectively solved, and meanwhile, the working efficiency and the service quality are greatly improved, so that even a non-professional operation and maintenance team can easily complete a complicated LLM deployment task only by inputting the LLM type and the requirement.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a configuration method, system, device, program product and storage medium. Background Art

[0002] With the widespread application of Large Language Model (LLM) in the field of natural language processing, the deployment and configuration of its reasoning service has become an important task for operation and maintenance personnel.

[0003] The working principle of the LLM model is complex, involving a large number of mathematical operations and data processing, and the demand for computing resources is also highly dynamic and diverse. During the reasoning process, the model needs to generate corresponding outputs based on the input text. This process involves multiple links such as model loading, reasoning calculation, and memory management. Each link has different requirements for computing resources. For example, the model loading stage may require a large number of disk write / output operations, while the reasoning calculation stage may consume a large amount of central processing unit (CPU) and graphics processing unit (GPU) resources.

[0004] However, since operation and maintenance personnel are not familiar with the working principle of the LLM model and its resource requirements, they often find it difficult to make the best choice when configuring inference services. On the one hand, if the resource configuration is too low, the service performance may be degraded and the user's real-time and accuracy requirements may not be met; on the other hand, if the resource configuration is too high, although the service performance can be ensured, it will increase unnecessary hardware expenses and cause a waste of resources. Summary of the invention

[0005] Based on the above problems, the present application provides a configuration method, system, device, program product and storage medium.

[0006] The embodiments of the present application disclose the following technical solutions:

[0007] A first aspect of an embodiment of the present application provides a configuration method, including:

[0008] Get the large language model LLM type and requirement information;

[0009] generating configuration parameters based on the LLM type and the requirement information;

[0010] Creating a test service based on the configuration parameters;

[0011] Perform stress testing on the test service using the test script to obtain stress testing results;

[0012] Based on the stress test results, the configuration parameters are adjusted to obtain configuration parameters that meet preset performance requirements.

[0013] In a possible implementation, performing a stress test on the test service using a test script to obtain a stress test result includes:

[0014] Using monitoring scripts to collect performance indicators generated during the operation of the test service;

[0015] For each performance indicator, the performance indicator is compared with the corresponding preset performance condition to obtain a performance indicator that does not meet the preset performance condition as a stress test result.

[0016] In a possible implementation, for each performance indicator, comparing the performance indicator with a corresponding preset performance condition, and obtaining a performance indicator that does not meet the preset performance condition as a stress test result includes:

[0017] For each performance indicator, the performance indicator is compared with the corresponding preset expansion condition and preset reduction condition to obtain a performance indicator that meets the preset expansion condition and / or meets the preset reduction condition as a stress test result. The stress test result is used to indicate whether to perform an expansion operation or a reduction operation on the configuration parameters corresponding to the performance indicator.

[0018] In a possible implementation, after creating the test service based on the configuration parameters, the method further includes:

[0019] Setting a test environment corresponding to the test service based on the configuration parameters;

[0020] The stress test of the test service using the test script to obtain the stress test result includes:

[0021] In the set test environment, the test service is stress tested using a test script.

[0022] In a possible implementation, the step of performing a stress test on the test service using a test script and obtaining a stress test result further includes:

[0023] If the requirement information indicates a target application scenario, a test script corresponding to the target application scenario is called.

[0024] In a possible implementation, the step of performing a stress test on the test service using a test script and obtaining a stress test result further includes:

[0025] If the requirement information does not indicate an application scenario, calling a script template whose matching degree with the requirement information meets a preset matching standard from a pre-built script library;

[0026] The script template is adjusted based on the requirement information to obtain a test script corresponding to the requirement information.

[0027] A first aspect of an embodiment of the present application provides a configuration system, including:

[0028] An acquisition unit, used to acquire LLM type and requirement information;

[0029] A generating unit, configured to generate configuration parameters based on the LLM type and the requirement information;

[0030] A creation unit, used for creating a test service based on the configuration parameters;

[0031] The testing unit is used to perform stress testing on the test service using a test script to obtain stress testing results;

[0032] The adjustment unit is used to adjust the configuration parameters based on the stress test results to obtain configuration parameters that meet preset performance requirements.

[0033] A third aspect of an embodiment of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the configuration method described in the first aspect is implemented.

[0034] A fourth aspect of the embodiments of the present application provides a computer program product. When the computer program product runs on a computer, the computer executes the configuration method as described in the first aspect above.

[0035] A fifth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions. When the instructions are executed on a terminal device, the terminal device executes the configuration method as described in the first aspect above.

[0036] Compared with the prior art, this application has the following beneficial effects:

[0037] By collecting LLM types and specific user demand information, it is ensured that subsequent operations have clear goal orientation. Based on the obtained LLM types and specific user demand information, the initial configuration parameters are generated. This method avoids the errors that may be caused by manual configuration, and also takes into account the best match between different hardware resources, thereby improving the success rate and rationality of the configuration. Use the generated configuration parameters to create a temporary test service. This test service is used to simulate the actual production environment for subsequent stress testing. The test service is stress tested using a test script to obtain stress test results, simulating the load conditions under extreme conditions, so as to verify stability and responsiveness, thereby identifying those performance bottlenecks that are not easy to detect under normal circumstances but affect user experience. According to the stress test results fed back by the test, the configuration parameters are adjusted until the pre-set performance standards are met, ensuring that the configuration parameters finally adjusted can achieve high performance and high stability, and meet user needs. This application effectively solves the resource configuration problem caused by lack of professional knowledge, and greatly improves work efficiency and service quality, so that even non-professional operation and maintenance teams can easily complete complex LLM deployment tasks by only entering LLM types and requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0039] Figure 1 A flowchart of a configuration method provided in an embodiment of the present application;

[0040] Figure 2 A schematic diagram of the deployment process provided for an embodiment of the present application;

[0041] Figure 3 A configuration system structure diagram provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0043] To facilitate understanding of the technical solutions provided by the embodiments of the present application, the professional terms involved in the embodiments of the present application will be explained below.

[0044] LLM inference service instance refers to one or more server instances that deploy large language models (LLM) and are used to perform inference tasks. These instances can be physical machines, virtual machines, or containerized environments. They are responsible for receiving user requests, processing input data by loading trained LLMs, and returning corresponding output results.

[0045] To facilitate understanding of the technical solutions provided by the embodiments of the present application, the background technology involved in the embodiments of the present application will be described below.

[0046] As mentioned above, in actual operations, operation and maintenance personnel often lack a deep understanding of the LLM reasoning characteristics, which directly leads to irrational configuration decisions, which in turn affects service performance and cost control. Specifically, due to the lack of targeted professional knowledge, it is difficult for operation and maintenance personnel to accurately evaluate the optimal parameter settings in different application scenarios, making the system configuration either too conservative, wasting resources, or too radical, with risks.

[0047] In addition, the current mainstream large model service platform mainly relies on traditional hardware resource monitoring indicators, such as CPU usage and memory usage, to judge the system load when implementing service expansion and contraction, and adjust the service level based on this. Although this approach can reflect the operating status of the system to a certain extent, it fails to fully consider the unique attributes of large model reasoning services. For example, key performance indicators such as service throughput, concurrent processing capabilities, output text length range, and expected generated content length range are not taken into consideration. These factors are critical to ensuring the response speed, stability, and user experience of the reasoning service. Because the existing mechanism fails to accurately capture and serve the above characteristics, the effectiveness of the expansion and contraction strategy is insufficient, which may not only cause unnecessary consumption of resources, but also affect the consistency and reliability of service quality.

[0048] In order to solve the above problems, the embodiment of the present application can provide personalized configuration parameters for different types of large language models (LLMs) by obtaining the type and specific demand information of LLMs. This helps to ensure that each LLM can run in the best state in its specific application scenario, thereby improving its performance. Automatically generating configuration parameters based on the acquired LLM type and demand information reduces the errors that may be caused by manual configuration and makes the configuration process more efficient and faster. At the same time, this method can also flexibly adjust the configuration according to different needs, enhancing the adaptability of the configuration process. Creating a test service and using a test script for stress testing can verify the validity and stability of the configuration parameters. The adjustment of the configuration parameters is guided by the actual stress test results, ensuring that the configuration parameters finally determined are the best choices that have been empirically tested and meet the preset performance requirements. This method not only improves the accuracy of the configuration, but also improves the reliability and efficiency of the configuration process in a real environment.

[0049] The final configuration parameters are obtained after repeated optimization based on the stress test results, so it can be ensured that LLM can achieve the expected performance indicators in actual applications. This is especially important for tasks that require high-performance computing resources, such as natural language processing, image recognition and other complex application scenarios. Traditional configuration methods often rely on professionals and require multiple attempts to find the most suitable settings. This method greatly reduces the time and resource consumption in this process by introducing automated testing and adjustment processes, and improves the efficiency of configuration work.

[0050] It should be noted that the configuration method, system, device and medium provided in the present application can be applied to the field of computer technology. The above is only an example and does not limit the application field of the configuration method, system, device and medium provided in the present application. In addition, the embodiments of the present application may not limit the execution subject of the configuration. For example, the configuration method of the embodiment of the present application can be applied to data processing devices such as terminal devices or servers. Among them, the terminal device can be an electronic device such as a computer, a personal digital assistant (PDA). The server can be a stand-alone server, a cloud server, or a cluster server composed of multiple servers.

[0051] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0052] The following describes a configuration method provided by the present application through an embodiment. Figure 1 ,Should Figure 1 A flowchart of a configuration method provided in an embodiment of the present application, the method comprising:

[0053] S101. Obtain LLM type and requirement information.

[0054] In actual application scenarios, a user interface or API interface can be designed to receive user-specified LLM types and requirement information. Users enter the LLM type to be deployed through the interface, or a series of predefined LLM options are provided on the interface, and users can select a model that suits their needs from a series of predefined LLM options. These options can be classified according to the characteristics of the model, such as language support, domain expertise (such as law, medical, etc.), model size, etc. This step ensures that targeted services can be provided according to different model requirements in the future.

[0055] The user inputs the demand information. In an actual application scenario, a structured form can be provided on the user interface for the user to input the demand information. The demand information is used to represent the user's specific requirements for the service.

[0056] The requirement information includes but is not limited to: Delay requirements: Time To First Token (TTFT) and Time Between Tokens (TBT). Users can set the expected maximum or average value to meet real-time or response speed requirements. Concurrency: Users need to specify the range of the number of requests expected to be processed simultaneously. This is critical for evaluating server resource requirements. Output length range: Users should be able to specify the minimum and maximum expected length of the generated text to ensure that the system does not generate results that are too short or too long. Expected generation length range: Indicate the maximum number of tokens you want the model to generate in each request, which has a direct impact on the allocation of computing resources.

[0057] After the user enters the LLM type and requirement information, the obtained LLM type and requirement information are verified to ensure that all required fields are correctly filled in and the values ​​are within a reasonable range. This process may include at least one of required field checks, data type checks, value range restrictions, and logical consistency checks, wherein the required field check is used to determine which fields must be filled in by the user and set default values ​​or prompt information for these fields. The data type check is used to confirm whether the data type of each field is correct (for example, numbers, strings, etc.) and convert the format as needed. The value range restriction is used to verify fields involving specific values, such as delay requirements, concurrent numbers, output length intervals, etc., and reasonable minimum and maximum values ​​should be set. The logical consistency check is used to ensure that the relationship between different fields is reasonable, such as the maximum generated length cannot be less than the maximum input length.

[0058] During the user input process, you can provide immediate feedback to help users correct mistakes during the input process. For example, when the user enters a value that is out of range, immediately display a warning message and suggest appropriate adjustments. Use friendly UI elements such as color changes, icon prompts, etc. to intuitively indicate the input status (correct / wrong).

[0059] In a possible implementation, before the user completes entering the LLM type and requirement information and attempts to submit the form, a complete validation process is performed to ensure that the LLM type and requirement information entered by the user meets all conditions. If any problems are found, the submission action is blocked and an alarm instruction is generated, and the problematic fields are highlighted, while clear guidance information is given.

[0060] During the verification process, TTFT (delay to first output) and TBT (time interval between two tokens) should be a positive integer or floating point number in milliseconds (ms) or seconds (s). A reasonable maximum value can be set according to the actual situation, such as TTFT not exceeding 500ms and TBT not exceeding 100ms. The number of concurrent requests should be a non-negative integer. Generally, this value should not exceed the upper limit of the system's processing capacity. A reasonable maximum number of concurrency can be set based on the capabilities of the target platform. The upper and lower limits of the output length interval should be non-negative integers, and the lower limit should not be greater than the upper limit. Considering model performance and user experience, a global maximum output length limit may also need to be set. For selective fields such as LLM type, verify whether the option selected by the user is valid; for text fields, regular expressions can be used for format verification. For the output length interval, the number of tokens in a single request can be verified, and the value can be verified to be a non-negative integer and within the model's capabilities.

[0061] In one possible implementation, in order to help users better understand the meaning and impact of each parameter, a brief help document or tool tip can be added to the interface. For users who are not sure how to fill in certain parameters, some recommended values ​​based on common scenarios can be provided as a reference.

[0062] As a result, the LLM type and requirement information entered by the user can be effectively verified, thereby improving data quality and reducing service problems caused by improper configuration. In addition, good user experience design can also help reduce the user's operational difficulty and improve overall satisfaction.

[0063] S102: Generate configuration parameters based on the demand information.

[0064] According to the specific user needs represented by the demand information, combined with the actual system hardware and software conditions, a detailed performance evaluation is performed to generate a series of service deployment configuration parameters. Factors that need to be considered in this step include but are not limited to model size, computing resource availability, network bandwidth, etc.

[0065] For example, the following configuration parameters can be determined based on the demand information:

[0066] Number of parallel cards: Based on the predicted number of concurrent requests and the computing resources required for each request, decide how many NPU devices to use, and set worldSize to indicate the total number of nodes in the distributed training / inference environment.

[0067] Maximum context length: Taking into account the different requirements of different applications for context dependence, a reasonable maximum context length is set in combination with the expected generation length range to determine the maximum context length.

[0068] Maximum generation length: Determine the longest text sequence that may be generated in each generation operation based on the requirements of the application scenario, especially the upper limit of the output length range.

[0069] Maximum input length: Set this parameter based on the average length and variation range of input text in actual applications to ensure that the model does not exceed the memory limit when processing long text.

[0070] Maximum batchsize: Optimize the batch size to balance efficiency and responsiveness based on hardware resources and latency requirements. maxPrefillBatchSize refers to the maximum batch size in the prefill phase, while maxBatchSize refers to the maximum batch size used in the decoding phase.

[0071] Based on the above evaluation results, automatically generate JSON or YAML format configuration files containing all necessary configuration items. These configuration files should clearly define the specific values ​​of each parameter and can be directly applied to the target environment. In actual application scenarios, a visualization tool or report can be provided to users to view the configuration parameters to be implemented, allowing users to make modification comments or adjustment suggestions. For any changes, the impact on overall performance should be re-evaluated.

[0072] S103: Create a test service based on the configuration parameters.

[0073] Use the configuration parameters determined in the previous step to build an LLM inference service in a development or test environment. Ensure that the service can correctly load the pre-trained LLM and provide inference capabilities as expected.

[0074] In actual application scenarios, the environment is prepared first. According to the hardware requirements in the service configuration (such as the number of parallel cards), select or build a test environment that is as similar as possible to the production environment. The selected test environment can include virtual machines, container clusters, or local laboratory environments on the cloud platform. All necessary software packages and services are installed on the selected test environment, including but not limited to the operating system, programming language runtime, machine learning framework, and any specific application dependencies.

[0075] Use the service configuration parameters generated in step S102 to write or adjust the existing service code to ensure that it can correctly parse the configuration file and initialize and start the service according to the specified parameter values. Integrate the configuration file into the service so that the configuration file is automatically loaded when the service starts and starts according to the specified parameters. The integration process can be implemented through environment variables, configuration management tools (such as Consul, etcd) or directly defined in the configuration file.

[0076] Select an appropriate deployment method (such as Docker container, Kubernetes cluster, etc.), and set the corresponding environment variables or configuration mapping according to the configuration parameters. Start the service to ensure that it can correctly load the configuration parameters and run normally.

[0077] In one possible implementation, a monitoring script or tool may be configured to collect and analyze various performance indicators of the test service in real time, such as response time, throughput, resource utilization, etc., to help analyze the cause of the problem afterwards.

[0078] In a possible implementation, after the test service is started, preliminary operations may be performed to warm up the service. For example, a small number of standard requests may be sent to verify whether the system is in a stable and responsive state when the test starts.

[0079] S104: Use the test script to perform stress testing on the test service to obtain stress testing results.

[0080] Write or select appropriate test scripts to simulate real request traffic, perform stress testing on the created test service, simulate actual workload, record various performance indicators and service performance during the test, and obtain stress testing results.

[0081] In a possible implementation, performing a stress test on the test service using a test script to obtain a stress test result includes:

[0082] The monitoring script is used to collect the performance indicators generated during the operation of the test service; for each performance indicator, the performance indicator is compared with the corresponding preset performance condition, and the performance indicator that does not meet the preset performance condition is obtained as the stress test result.

[0083] Start the stress test and gradually increase the load until the target value is reached. Keep the load stable for a while during the whole process to observe the performance of the system under high load.

[0084] Use monitoring scripts to collect various performance indicators of the service in real time. For each collected performance indicator, there is one or a group of preset performance conditions as a reference standard, such as the maximum allowed response time, the minimum success rate, etc. Compare the actual performance indicators with the preset conditions one by one to identify those that fail to meet the requirements. Sorting out the performance indicators that do not meet the preset performance conditions as the stress test results.

[0085] In a possible implementation, for each performance indicator, comparing the performance indicator with a corresponding preset performance condition, and obtaining a performance indicator that does not meet the preset performance condition as a stress test result includes:

[0086] For each performance indicator, the performance indicator is compared with the corresponding preset expansion condition and preset reduction condition to obtain a performance indicator that meets the preset expansion condition and / or meets the preset reduction condition as a stress test result. The stress test result is used to indicate whether to perform an expansion operation or a reduction operation on the configuration parameters corresponding to the performance indicator.

[0087] Expansion means adding resources to cope with higher load requirements. Expansion operations can include increasing the number of servers, improving server specifications, adding high-performance computing resources such as GPU / TPU, etc. For example, it can be triggered when the following situations are detected: the response time is too long, that is, when the monitoring script detects that the average response time exceeds the preset threshold, which indicates that the current resources are not enough to quickly process all requests. At this time, you should consider increasing the number of server instances or allocating more computing resources. Excessive resource utilization, that is, if the CPU, memory or GPU usage is maintained at a very high level for a long time (such as more than 90%), it means that the system may be overloaded and more resources are needed to share the pressure. The error rate rises, that is, when the service error rate increases significantly, it may be due to service instability caused by insufficient resources. In this case, expansion can help alleviate the problem. The queue length increases, that is, if the number of requests waiting to be processed increases sharply, forming a longer queue, which is also an obvious signal for expansion.

[0088] For example, select a representative benchmark prompt that reflects the characteristics of typical user requests. Run the inference service under this prompt and record key performance indicators, such as TTFT (Total Time to First Token), TBT (Token By Token), and call error rate. These benchmark indicators will serve as a reference for subsequent performance monitoring and expansion decisions.

[0089] Monitor the above indicators in real time and set reasonable thresholds. For example, if TBT increases by more than 50%, or TTFT exceeds 1 second, this may mean that the service is facing a performance bottleneck. At the same time, you also need to pay attention to the call error rate. If the error rate exceeds 3%, this may indicate that there are unstable factors in the service, which need to be promptly checked and considered for expansion.

[0090] In addition to performance indicators, you can also monitor the system load. The load can reflect the pressure level of the service instance when processing requests. If the system load average value continues to exceed 1.5 times that of a single service instance, this may mean that the current number of service instances can no longer meet demand and you need to consider expanding capacity.

[0091] The specific thresholds in the above examples (such as 50%, 1 second, 3%, 1.5 times, etc.) need to be determined based on business needs and historical data. These thresholds are used to reflect the performance bottlenecks and load pressure of the service so as to trigger the expansion operation at the appropriate time. At the same time, these thresholds can also be dynamically adjusted according to the development of the service. For example, as the number of users increases or the complexity of the business increases, it may be necessary to lower certain thresholds to trigger the expansion in advance.

[0092] Once the expansion condition is triggered, the expansion process is executed. The expansion process may include:

[0093] Based on monitoring data and business needs, evaluate the number of service instances that need to be expanded. Prepare sufficient computing resources (such as servers, GPUs, etc.) to support the expansion. Deploy LLM inference service instances on the new computing resources. Configure the load balancer to evenly distribute requests to the new service instances. Continue to monitor the performance of the expanded service and make adjustments as needed.

[0094] Scaling down means reducing resources to save costs or optimize resource allocation. Scaling down operations may include reducing the number of servers, reducing server specifications, releasing computing resources that are no longer needed, etc. Exemplary, it may be triggered when the following situations are detected: Low load period: During non-peak hours, if monitoring data shows that resource utilization has dropped significantly (for example, below 30%), then resource allocation can be appropriately reduced to avoid waste. Long-term idleness: If an instance or a group of instances is in a low-activity state for a long time, and there are almost no tasks queued for processing, then a scaling down operation can be used to reduce the scale.

[0095] S105. Adjust the configuration parameters based on the stress testing result.

[0096] Based on the stress test results, the configuration parameters obtained in the previous steps are fine-tuned to optimize performance. This ensures that the final service configuration is verified and optimized, and is more in line with the actual workload.

[0097] In actual application scenarios, you can choose appropriate stress testing tools (such as Apache JMeter, Locust, k6, etc.) according to service characteristics and expected load. In addition, you can write or adjust existing stress testing scripts to simulate expected user behavior and traffic patterns. This includes but is not limited to the number of concurrent users, request frequency, request content, etc.

[0098] During stress testing, you can first perform a small round of tests to verify that the test script works as expected. Start with a low load and gradually increase the number of concurrent requests until you reach or exceed the expected maximum load. Try to exceed the maximum expected load to evaluate the stability and recovery capabilities of the service.

[0099] Analyze the data collected from monitoring tools, such as response time, throughput, error rate, etc., to evaluate the performance of the service. Find the cause of the performance degradation, which may be due to resource limitations (CPU, memory), network latency, or other factors. Check the service logs for abnormal behavior or error information.

[0100] In a possible implementation, all collected data can be organized into an easy-to-understand form, such as charts, tables, etc. A stress test report is written based on the collected data, summarizing the problems found during the test, solution suggestions, and performance improvement after optimization. The configuration parameters are adjusted based on the stress test report, and the above test process is repeated until the performance requirements are met.

[0101] In a possible implementation, step S105 may further include:

[0102] S106. Configure the LLM reasoning service instance based on the adjusted configuration parameters.

[0103] Configure the LLM reasoning service instance in the production environment using the fine-tuned optimal service configuration. Ensure that all configurations meet the preset performance requirements and are ready to handle actual business requests.

[0104] In actual application scenarios, once the LLM reasoning service is successfully launched, continuous monitoring can be performed to ensure its stability and efficiency.

[0105] In actual application scenarios, Prometheus can be used as the main time series database, and the Grafana visualization panel can be used to display the changing trends of various indicators. At the same time, OpenTelemetry can be used to collect distributed tracing information.

[0106] According to the characteristics of large model services, standard conditions for measuring service quality are established, such as throughput, latency, accuracy, etc. In particular, for different application scenarios (such as customer service chatbots vs. medical diagnosis assistants), key indicators that best reflect the user experience in the scenario can be adaptively selected. For each indicator that meets the preset critical conditions, a corresponding threshold range is set, and an instant notification mechanism (such as email, SMS, voice reminder, etc.) is configured to respond to abnormal conditions in a timely manner.

[0107] When it is detected that the utilization of some resources is close to the limit, the expansion operation is automatically triggered; on the contrary, the reduction strategy is implemented during the off-peak period to save costs. Specifically, it can be comprehensively judged whether the instance scale needs to be adjusted based on multiple dimensions such as CPU utilization, memory occupancy, disk I / O speed, etc. The specific process is described in the expansion operation and reduction operation execution strategy in the above step S104, which will not be repeated here.

[0108] In one possible implementation, historical data can be reviewed regularly to find possible bottlenecks or optimization opportunities. For example, by comparing performance differences in different time periods, it is possible to identify time periods with higher load fluctuations, and then develop more targeted response plans.

[0109] The following describes the configuration process based on the deployment of the user interface and the LLM backend. Figure 2 , Figure 2 A schematic diagram of the deployment process provided for an embodiment of the present application.

[0110] First, the user enters the characteristics of the application scenario through the configuration interface. These characteristics will be used as input parameters for subsequent automated deployment. The LLM deployment backend system generates the corresponding MindIE (inference service) configuration parameters based on the application scenario characteristics entered by the user. Based on the generated configuration parameters, the LLM deployment backend system starts a test service instance. Test the started test service to collect performance data and indicators. Analyze the test results and adjust the MindIE configuration parameters based on the test data to optimize service performance. Use the optimized configuration parameters to formally deploy the inference service. After deployment, the system continuously monitors the operating quality of the service, including key indicators such as response time and throughput.

[0111] If the monitoring results show that the service quality does not meet the preset conditions (for example, the response time is too long or the throughput is insufficient), the expansion mechanism is triggered to increase the service nodes to improve the service capacity. After the service deployment is completed, the service is provided through the inference API, and users can call the inference service through the API interface.

[0112] Expanded to Figure 2 A decision point in the process is to decide whether to expand capacity based on the service quality monitoring results. If the service quality meets the requirements, the process ends; if not, the expansion operation is triggered.

[0113] In summary, this application has the following beneficial effects:

[0114] By generating deployment parameters in an automated and intelligent way, deployment personnel do not need to have an in-depth understanding of complex system architecture and configuration details, but only need to focus on the needs of actual application scenarios. This greatly reduces the requirements for deployment personnel's professional knowledge, greatly broadens the scope of personnel who can perform deployment operations, and reduces dependence on professional technicians.

[0115] Adjust configuration parameters based on test results to ensure that each modified configuration can quickly get test result feedback, so that users can find the optimal configuration solution in the shortest time, thus speeding up the process from development to launch. Automatically adjust parameters to meet different needs, reduce the number of manual configurations and tests, thereby reducing test failures and repeated testing workloads caused by incorrect or inappropriate configurations. Dynamically adjust the number and configuration of service nodes based on real-time service needs and performance indicators. This avoids waste and idleness of resources and improves resource utilization and efficiency.

[0116] The above are some specific implementations of the configuration method provided in the embodiment of the present application. Based on this, the present application also provides a corresponding configuration system. The system provided in the embodiment of the present application will be introduced from the perspective of functional modularization. Figure 3 A configuration system structure diagram provided in an embodiment of the present application.

[0117] The system comprises:

[0118] An acquisition unit 110 is used to acquire a large language model LLM type and requirement information;

[0119] A generating unit 111, configured to generate configuration parameters based on the LLM type and the requirement information;

[0120] A creating unit 112, configured to create a test service based on the configuration parameters;

[0121] The testing unit 113 is used to perform a stress test on the test service using a test script to obtain a stress test result;

[0122] The adjusting unit 114 is used to adjust the configuration parameters based on the stress test results to obtain configuration parameters that meet preset performance requirements.

[0123] The embodiments of the present application also provide corresponding devices and computer storage media for implementing the configuration method provided in the embodiments of the present application.

[0124] The device includes a memory and a processor, the memory is used to store instructions or codes, and the processor is used to execute the instructions or codes so that the device executes the configuration method described in any embodiment of the present application.

[0125] The computer storage medium stores codes, and when the codes are executed, a device executing the codes implements the test method described in any embodiment of the present application.

[0126] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the system or device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0127] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0128] It should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0129] It should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0130] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0131] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0132] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A configuration method, characterized in that: include: Get the LLM type and requirement information; generating configuration parameters based on the LLM type and the requirement information; Creating a test service based on the configuration parameters; Perform stress testing on the test service using the test script to obtain stress testing results; Based on the stress test results, the configuration parameters are adjusted to obtain configuration parameters that meet preset performance requirements.

2. The method according to claim 1, characterized in that The stress test of the test service using the test script to obtain the stress test result includes: Using monitoring scripts to collect performance indicators generated during the operation of the test service; For each performance indicator, the performance indicator is compared with the corresponding preset performance condition to obtain a performance indicator that does not meet the preset performance condition as a stress test result.

3. The method according to claim 2, characterized in that For each performance indicator, the performance indicator is compared with the corresponding preset performance condition to obtain a performance indicator that does not meet the preset performance condition as a stress test result, including: For each performance indicator, the performance indicator is compared with the corresponding preset expansion condition and preset reduction condition to obtain a performance indicator that meets the preset expansion condition and / or meets the preset reduction condition as a stress test result. The stress test result is used to indicate whether to perform an expansion operation or a reduction operation on the configuration parameters corresponding to the performance indicator.

4. The method according to claim 1, characterized in that: After creating the test service based on the configuration parameters, the method further includes: Setting a test environment corresponding to the test service based on the configuration parameters; The stress test of the test service using the test script to obtain the stress test result includes: In the set test environment, the test service is stress tested using a test script.

5. The method according to claim 1, characterized in that: Before performing stress testing on the test service using the test script and obtaining the stress testing result, the method further includes: If the requirement information indicates a target application scenario, a test script corresponding to the target application scenario is called.

6. The method according to claim 1, characterized in that Before performing stress testing on the test service using the test script and obtaining the stress testing result, the method further includes: If the requirement information does not indicate an application scenario, calling a script template whose matching degree with the requirement information meets a preset matching standard from a pre-built script library; The script template is adjusted based on the requirement information to obtain a test script corresponding to the requirement information.

7. A configuration system, characterized in that: include: An acquisition unit, used to acquire LLM type and requirement information; A generating unit, configured to generate configuration parameters based on the LLM type and the requirement information; A creation unit, configured to create a test service based on the configuration parameters; The testing unit is used to perform stress testing on the test service using a test script to obtain stress testing results; The adjustment unit is used to adjust the configuration parameters based on the stress test results to obtain configuration parameters that meet preset performance requirements.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the configuration method according to any one of claims 1 to 6 is implemented.

9. A computer program product, characterized in that When the computer program product runs on a computer, the computer executes the configuration method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on the terminal device, the terminal device executes the configuration method according to any one of claims 1 to 6.

Images