A model deployment method and device, electronic equipment and storage medium

Abstract

This application provides a model deployment method, apparatus, electronic device, and storage medium. The method includes: receiving user instructions; invoking a large language model to obtain document information of the model to be deployed, hardware resource information of the target computing environment, and deployment framework information; obtaining historical running data; fusing multimodal features of this data and assembling it into dynamic prompt words; invoking the large language model to generate deployment parameters; verifying the deployment parameters through a lightweight simulator; if the verification is successful, generating a deployment file and invoking the target program to perform containerized deployment; if the deployment is successful, performing performance testing and comparing the test performance indicators with the historical best performance indicators; if the deviation of the comparison exceeds a preset threshold or the deployment fails, inputting the deployment log into the reinforcement learning model. The embodiments of this application achieve intelligent, adaptive, and closed-loop optimization of model deployment through the above method.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and more specifically, to a model deployment method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the rapid development of artificial intelligence technology, complex neural network models such as large language models and image generation models are widely used in various industries. The process of migrating these models from the R&D environment to the production environment, i.e., model deployment, is a key step in realizing their commercial value. Model deployment requires configuring dozens or even hundreds of parameters (such as context length, tensor parallelism, batch size, memory utilization threshold, etc.) based on the hardware resources of the target computing environment (such as GPU model, memory size, CPU core count, and memory capacity) and the characteristics of the deployment framework, to ensure that the model can run stably and fully utilize the hardware performance.

[0003] Currently, the configuration of model deployment parameters mainly relies on manual experience or static scripts. Research has revealed that existing deployment methods suffer from several problems: deployment parameters depend on manual experience for configuration, there is a lack of dynamic adjustment mechanisms, slow response after deployment failures, and an inability to learn from historical deployments to continuously optimize performance. Summary of the Invention

[0004] In view of this, embodiments of this application provide a model deployment method, apparatus, electronic device, and storage medium to achieve intelligent, adaptive, and closed-loop optimization of model deployment.

[0005] In a first aspect, embodiments of this application provide a model deployment method, the method comprising: Receive instructions from users to deploy the model to the target computing environment; The system acquires the document information of the model to be deployed, the hardware resource information of the target computing environment, and the preset deployment framework information; and acquires historical running data from the historical deployment database; the historical running data includes the environment information, deployment parameters, performance indicators, and success / failure status of the model to be deployed in the past. The document information, hardware resource information, deployment framework information, and historical operation data are fused using multimodal features. Based on the fused data, dynamic prompt words are automatically assembled, and a large language model is invoked to generate deployment parameters according to the dynamic prompt words. The deployment parameters are verified using a lightweight simulator to determine whether they meet the deployment conditions of the model to be deployed. If the verification passes, a deployment file is generated based on the deployment parameters, and the target program is invoked to perform containerized deployment. If deployment is detected as successful, a performance test is performed to obtain the current performance metrics, and the current performance metrics are compared with the historical best performance metrics corresponding to the hardware resource information in the historical deployment database. If the deviation of the comparison result exceeds the preset threshold or a deployment failure is detected, the deployment log is input into the reinforcement learning model. The reinforcement learning model evaluates the deployment according to the preset reward function and outputs the parameter adjustment direction, triggering the large language model to update the deployment parameters according to the parameter adjustment direction, until the deviation of the comparison result does not exceed the preset threshold or the preset number of retries is reached.

[0006] In one feasible implementation, the method further includes: The hardware resource information, deployment parameters, performance indicators, and success / failure status of this deployment are stored in the historical deployment database.

[0007] In a feasible implementation, the reinforcement learning model employs a proximal policy optimization algorithm, which generates parameters to adjust the direction through a policy network, evaluates the quality of the adjusted direction through a value network, and limits the magnitude of each parameter adjustment.

[0008] In one feasible implementation, the state space of the reinforcement learning model includes the hardware resource information, current deployment parameters, vector embedding of deployment logs, and success or failure cases in the historical deployment database. The reward function is used to assign a positive reward when deployment is successful, and to add an additional reward based on the ratio of actual throughput to target throughput; to assign a negative reward when deployment fails, and to add an additional penalty when resource utilization is lower than a preset threshold.

[0009] In one feasible implementation, the lightweight simulator is a pre-built video memory calculation algorithm used to estimate video memory usage and determine whether the deployment parameters will cause video memory overflow based on the number of parameters of the model to be deployed, the number of graphics cards, and the size of video memory per card.

[0010] In one feasible implementation, the performance test includes concurrent stress testing, and the current performance metric includes at least one of requests processed per second, average response latency, and success rate; The historical best performance metric refers to the highest number of requests per second or the lowest response latency achieved in a historical deployment under the same hardware resource information.

[0011] In one feasible implementation, the automatic assembly of the dynamic prompt words is accomplished through the following steps: The model parameter quantity and quantization support information in the document information, the graphics card model, number of graphics cards, video memory size, memory size, number of central processing unit cores in the hardware resource information, and the successful deployment parameter cases in the historical running data are structurally spliced ​​together according to the preset prompt word template.

[0012] Secondly, embodiments of this application also provide a model deployment apparatus, the apparatus comprising: The receiving module is used to receive instructions from users to deploy the model to the target computing environment. The acquisition module is used to acquire the document information of the model to be deployed, the hardware resource information of the target computing environment, and the preset deployment framework information; and to acquire historical running data from the historical deployment database; the historical running data includes the environment information, deployment parameters, performance indicators, and success / failure status of the model to be deployed in the past. The prompt module is used to perform multimodal feature fusion of the document information, hardware resource information, deployment framework information and historical running data, automatically assemble dynamic prompt words based on the fused data, and call a large language model to generate deployment parameters according to the dynamic prompt words; The verification module is used to verify the deployment parameters through a lightweight simulator to determine whether the deployment parameters meet the deployment conditions of the model to be deployed. The deployment module is used to generate a deployment file based on the deployment parameters and call the target program to perform containerized deployment if the verification passes. The testing module is used to perform performance tests to obtain the current performance metrics if successful deployment is detected, and to compare the current performance metrics with the historical best performance metrics corresponding to the hardware resource information in the historical deployment database. An adjustment module is used to input deployment logs into a reinforcement learning model if the deviation of the comparison result exceeds a preset threshold or a deployment failure is detected. The reinforcement learning model evaluates the deployment based on a preset reward function and outputs a parameter adjustment direction, triggering the large language model to update the deployment parameters according to the parameter adjustment direction, until the deviation of the comparison result does not exceed the preset threshold or the preset number of retries is reached.

[0013] In one feasible implementation, the device further includes: The storage module is used to store the hardware resource information, deployment parameters, performance indicators, and success / failure status of this deployment into the historical deployment database.

[0014] In a feasible implementation, the reinforcement learning model employs a proximal policy optimization algorithm, which generates parameters to adjust the direction through a policy network, evaluates the quality of the adjusted direction through a value network, and limits the magnitude of each parameter adjustment.

[0015] In one feasible implementation, the state space of the reinforcement learning model includes the hardware resource information, current deployment parameters, vector embedding of deployment logs, and success or failure cases in the historical deployment database. The reward function is used to assign a positive reward when deployment is successful, and to add an additional reward based on the ratio of actual throughput to target throughput; to assign a negative reward when deployment fails, and to add an additional penalty when resource utilization is lower than a preset threshold.

[0016] In one feasible implementation, the lightweight simulator is a pre-built video memory calculation algorithm used to estimate video memory usage and determine whether the deployment parameters will cause video memory overflow based on the number of parameters of the model to be deployed, the number of graphics cards, and the size of video memory per card.

[0017] In one feasible implementation, the performance test includes concurrent stress testing, and the current performance metric includes at least one of requests processed per second, average response latency, and success rate; The historical best performance metric refers to the highest number of requests per second or the lowest response latency achieved in a historical deployment under the same hardware resource information.

[0018] In one feasible implementation, the automatic assembly of the dynamic prompt words is accomplished through the following steps: The model parameter quantity and quantization support information in the document information, the graphics card model, number of graphics cards, video memory size, memory size, number of central processing unit cores in the hardware resource information, and the successful deployment parameter cases in the historical running data are structurally spliced ​​together according to the preset prompt word template.

[0019] Thirdly, embodiments of this application also provide an electronic device, including: a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the method as described in any one of the first aspects.

[0020] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the method as described in any one of the first aspects.

[0021] This application provides a model deployment method, apparatus, electronic device, and storage medium. After receiving a deployment instruction, the method acquires the document information, hardware resource information, and deployment framework information of the model to be deployed, and performs multimodal feature fusion by combining it with the running data in the historical deployment database. It automatically assembles dynamic prompt words to generate deployment parameters, and then performs containerized deployment after verification by a lightweight simulator. After successful deployment, performance testing is performed and compared with historical best performance indicators. If the performance does not meet the standards or the deployment fails, the deployment log is input into the reinforcement learning model for evaluation. The model's output parameters are adjusted to adjust the direction and trigger the language model to update its parameters until the performance meets the standards or the number of retries is reached.

[0022] Based on the above solution, this application integrates document understanding, parameter generation, resource verification, performance evaluation, and closed-loop optimization into a single system, achieving intelligent model deployment throughout the entire process. The large language model automatically generates initial parameters adapted to the current hardware environment and deployment framework through understanding and fusing multimodal information, eliminating the need for manual document reading and configuration, significantly reducing the deployment threshold and error probability. The lightweight simulator pre-verifies parameters before deployment, avoiding deployment failures due to inappropriate parameters. If performance is substandard or deployment fails after deployment, logs are automatically captured and evaluated using a reinforcement learning model. The reinforcement learning model outputs adjustments, and the large language model updates parameters and retryes, forming a self-healing closed loop, significantly improving fault response speed and deployment success rate. Simultaneously, experience data from each deployment is stored in a historical database for reference in subsequent deployments and model optimization, enabling this application embodiment to continuously learn from history and become increasingly accurate with use.

[0023] Compared with existing technologies that rely on manual experience and static configuration, and lack dynamic adjustment and historical reference, the embodiments of this application effectively solve the problems of low deployment efficiency, slow failure response, and difficulty in continuous performance optimization.

[0024] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0025] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A flowchart of a model deployment method provided in an embodiment of this application is shown.

[0027] Figure 2 A flowchart of another model deployment method provided by an embodiment of this application is shown.

[0028] Figure 3 A schematic diagram of the structure of a model deployment device provided in an embodiment of this application is shown.

[0029] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0031] In daily algorithm development and production operations, model deployment is an unavoidable step. From the time a model is trained in the laboratory to the time it is actually provided to the public, there is a significant hurdle to overcome.

[0032] Typically, the engineer responsible for deployment needs to first find the model's documentation to understand its size and any special requirements. Then, they need to log into the target server to check the graphics card model, video memory size, CPU, and RAM specifications. Next, they need to search online or guess based on experience to determine the appropriate parameters for the hardware and model. If using an unfamiliar framework, they must first learn the framework's parameter configuration rules. Sometimes the first deployment fails with an out-of-memory error, requiring the engineer to check logs, investigate the cause, modify parameters, and redeploy. Even if deployment succeeds, the speed and whether it fully utilizes the hardware's performance are often unknowns.

[0033] This scenario plays out daily in many teams. Whether it's algorithm engineers wanting to quickly validate model performance or operations personnel needing to deploy models to the production environment, they inevitably get bogged down by this cumbersome process. When faced with a new model, a new framework, or new hardware, previous experience may become irrelevant, and everything has to start from scratch.

[0034] Based on this, embodiments of this application provide a model deployment method, apparatus, electronic device, and storage medium. The method described in these embodiments is specifically designed to solve problems in such scenarios. It transforms the aforementioned cumbersome operations into a simple dialogue between the user and the big oracle model. The user only needs to tell the deployment tool "which model I want to deploy and which environment to use," and the remaining tasks of reading documentation, checking hardware, configuring parameters, running tests, and optimizing can be automatically completed by this method. In this way, deployment work that originally took half a day or even longer can be completed in minutes, and the results are often better than manual configuration. The following is a description through embodiments.

[0035] To facilitate understanding of this embodiment, a model deployment method disclosed in this application will first be described in detail. For example... Figure 1 As shown, the method includes the following steps: Step 101: Receive the user's instruction to deploy the model to be deployed to the target computing environment.

[0036] When a user needs to deploy a trained model (such as an image generation model) to a production environment, the first step is to deploy it to designated computing resources. In this step, the deployment tool receives a deployment request from the user, which can be triggered by a click on the user interface or by sending an instruction via an application programming interface (API).

[0037] The "target computing environment" here refers to the hardware and software environment in which the model will ultimately run. This could be a physical server, a virtual machine instance, or a Pod within a containerized platform. Users typically specify in their instructions which model they want to deploy and where they expect it to be deployed.

[0038] In practical applications, users can trigger commands in various flexible ways. A common method is for the user to select the model to be deployed and the target environment on the operation page, and then click the "Deploy" button. Another method is to call the deployment interface via command-line tools or scripts. In some scenarios, users can even directly make requests to the deployment tool through natural language dialogue. For example, a user might type in the dialogue interface, "Please deploy the image classification model I just trained, using the environment on my computer." The deployment tool understands the user's intent from the dialogue and parses the identifier of the model to be deployed and the target environment. If the user does not explicitly specify the computing environment, the default target computing environment can be the terminal environment currently running the dialogue service.

[0039] Step 102: Obtain the document information of the model to be deployed, the hardware resource information of the target computing environment, and the preset deployment framework information; and obtain historical running data from the historical deployment database; the historical running data includes the environment information, deployment parameters, performance indicators, and success / failure status of the model to be deployed in the past.

[0040] After receiving the instructions mentioned in step 101, the deployment tool needs to prepare for the generation of deployment parameters. The core of this step is to collect various necessary information.

[0041] The first step is to collect relevant information about the model to be deployed. The deployment tool can call the corresponding API to collect this information. This documentation is quite extensive, including the documentation accompanying the model release (commonly a README file in a technical community) and the technical specifications recorded in the Model Card. These documents typically specify the model's parameter count, its architectural features, officially recommended hardware requirements, supported quantization methods, and sometimes provide deployment suggestions. This information can be obtained by reading files from a specified storage path or by pulling them from a model repository using the model's unique identifier.

[0042] Next, it's necessary to understand the hardware resource information of the target computing environment. This includes the model and number of graphics cards, the video memory size of each graphics card, the number of CPU cores, and the available memory capacity. This information can be read by calling the interfaces provided by the operating system, or, in a containerized environment, from the container platform's metadata service. Understanding this hardware information is crucial because the same model can support different levels of concurrency and requires different parameter adjustments depending on the hardware configuration.

[0043] It's also necessary to know which framework the user intends to use to deploy the model. Currently, there are many mainstream inference frameworks, such as vLLM, TensorRT-LLM, and TGI. Different frameworks have slightly different definitions and parsing methods for parameters; the same parameter name may have different value ranges in different frameworks, or some frameworks may support certain features while others do not. Therefore, explicitly specifying the deployment framework is a prerequisite for generating valid parameters.

[0044] In addition to the information mentioned above, it's also necessary to query the deployment tool's historical deployment database. This database stores experience data accumulated from past deployments. For the same model to be deployed, or for models with similar architectures, the historical deployment database may contain information such as the environment during the previous deployment, the deployment parameters used, the performance metrics measured after deployment, and whether the deployment was ultimately successful or not. This historical data is like a record of previous attempts, providing valuable references for the current deployment, such as what parameter combinations have achieved good results on similar hardware, and what parameters are prone to causing failure.

[0045] Step 103: Perform multimodal feature fusion on the document information, hardware resource information, deployment framework information and historical operation data, automatically assemble dynamic prompt words based on the fused data, and call the large language model to generate deployment parameters according to the dynamic prompt words.

[0046] After collecting documentation, hardware resources, deployment framework information, and historical operational data, the deployment tool needs to transform this information into a "requirements specification" that the large language model can understand and provide professional advice.

[0047] This information comes in various forms. Documentation information might be lengthy text descriptions, such as the explanatory text in a README file; hardware resource information is structured data, such as "graphics cards: 4 A100 80G, memory: 512GB"; deployment framework information is the user-specified framework name; and historical runtime data is tabular data recording past experiences. Integrating these different forms of information requires multimodal feature fusion. The goal of fusion is to present this information in a unified and interconnected way, rather than simply piling it up. For example, hardware information can be mapped to recommended configurations mentioned in documentation, and parameters from historical success stories can be linked to the current situation.

[0048] After the integration is complete, the deployment tool automatically assembles a dynamic prompt word based on this information. This prompt word is not fixed but dynamically generated according to the specific circumstances of each deployment. The content of the prompt word tells the large language model the current task context: what kind of model exists, and what its characteristics are as understood from the documentation; what the hardware configuration of the target environment is; what framework the user specified for deployment; and historically, under similar conditions, which parameter combinations have succeeded and which parameters are prone to problems. Then, the prompt word makes a request in a format that the large language model is good at understanding, so that it can provide a set of reasonable deployment parameters based on this information.

[0049] The assembled dynamic prompts are fed into a large language model. This large language model is a model with extensive technical knowledge, understanding deployment-related concepts and parameter meanings. The deployment tool is responsible for calling this model, but the model itself does not directly interact with the user or perform deployment operations. Its role is to understand the background and requirements described in the dynamic prompts, leveraging its knowledge learned from massive amounts of technical documentation and case studies to generate a set of deployment parameters. These deployment parameters typically include key configurations for model runtime, such as context length, tensor parallelism, memory utilization limits, and whether certain acceleration modes are enabled. The generated deployment parameters are returned in a structured format, such as JSON, for easy parsing and use by the deployment tool.

[0050] In this way, the complex process that originally required manual reading of documents, understanding of hardware, review of historical cases, and decision-making based on experience is delegated to a large language model with extensive knowledge to assist in completing the task. The deployment tool plays the role of organizing information, calling the model, and parsing results, while the large language model plays the role of providing deployment parameters based on the information.

[0051] Step 104: Verify the deployment parameters using a lightweight simulator to determine whether the deployment parameters meet the deployment conditions of the model to be deployed.

[0052] After the large language model generates deployment parameters, these parameters still need to undergo a rapid verification process to determine their feasibility. The purpose of this step is to pre-check the deployment parameters using a lightweight simulator before actually starting the deployment, thus preventing obviously unreasonable parameter combinations that are bound to fail from entering the actual deployment process.

[0053] For example, the lightweight simulator has a pre-built video memory calculation algorithm, which is used to estimate video memory usage and determine whether the deployment parameters will cause video memory overflow based on the number of parameters of the model to be deployed, the number of graphics cards, and the size of video memory per card.

[0054] The lightweight simulator here is a standalone computing tool that can be used to estimate resource usage in various scenarios. For example, during model development, researchers can use it to estimate memory requirements under different configurations; during capacity planning, operations personnel can use it to determine whether existing hardware can support the new model. However, in this embodiment, the lightweight simulator is used to initially derive deployment parameters, serving as the first verification checkpoint before deployment.

[0055] The core function of a lightweight simulator is to quickly estimate the memory usage required for deployment with a certain set of parameters, based on the number of parameters and architectural characteristics of the model to be deployed, combined with information such as the number of graphics cards and the size of each card's video memory in the target computing environment. For example, for a model with 70 billion parameters, if deployed using tensor parallelism of 4, the approximate number of parameters to be loaded on each card and the amount of video memory required to store intermediate calculation results can be roughly estimated using empirical formulas or pre-built model video memory calculation algorithms.

[0056] After estimating the video memory usage, the lightweight emulator compares it with the actual available video memory in the target environment. If the estimated usage significantly exceeds the available video memory, for example, if the calculation shows that 40GB of video memory is needed, but each card only has 24GB, then this set of parameters is likely to cause an OutOfMemoryError (OOM) during deployment, and will be judged as not meeting the deployment conditions.

[0057] Besides video memory, lightweight simulators sometimes make rough estimates of other resources, such as whether CPU and memory requirements are within reasonable limits. However, in practical applications, video memory is often the most likely resource to become a bottleneck in model deployment, so lightweight simulator verification usually focuses on video memory usage.

[0058] If the verification passes, it means that the deployment parameters are feasible at the resource level, and we can proceed to the next step of actual deployment. If the verification fails, the deployment tool can choose to abandon the deployment parameters or trigger a subsequent optimization process to allow the large language model to regenerate a more conservative set of parameters.

[0059] In this way, the lightweight simulator plays an initial corrective role before deployment, filtering out parameter combinations that are obviously likely to fail, saving time from repeated attempts, and avoiding interference with the environment from repeated failures.

[0060] Step 105: If the verification is successful, generate a deployment file based on the deployment parameters and call the target program to perform containerized deployment.

[0061] After verification using a lightweight simulator, if the deployment parameters are confirmed to be feasible at the resource level, the next step is to actually begin deployment according to these parameters. The core of this step is to translate the abstract parameters into specific deployment commands and actually start the model service.

[0062] The deployment tool automatically generates a corresponding deployment file based on the deployment parameters. In containerized deployment scenarios, this file is typically in YAML format, serving as a "blueprint" that container orchestration platforms like Kubernetes can recognize. This file specifies many details, such as the number of container replicas to be launched, the number of GPUs, CPU cores, and memory allocated to each container, the commands to be run within the container, environment variable settings, network connectivity, and so on. Various configurations in the deployment parameters, such as context length, tensor parallelism, and memory utilization thresholds, are filled into the corresponding locations in this file, becoming the runtime parameters when the container starts.

[0063] After generating the deployment file, the deployment tool calls the target program to perform the actual deployment operation. This target program is usually a command-line tool or application programming interface provided by the containerization platform, such as kubectl for Kubernetes. The deployment tool submits the generated deployment file to this target program, which is responsible for parsing the file content and interacting with the underlying container runtime (such as Docker) and hardware drivers (such as the NVIDIA Container Toolkit), ultimately launching the model service container in the target computing environment.

[0064] Once the container starts, the deployed (or undeployed) model begins running, waiting to receive external inference requests. During this process, the deployment tool continuously monitors the container's running status, such as whether the container started successfully, whether it exited abnormally, and whether there are error messages in the logs. This monitoring information provides a basis for subsequent steps (such as retrying on failure and performance testing).

[0065] Through this step, the numbers that were originally just stored in the parameter list are transformed into a truly running model service. The entire process is completed automatically by the deployment tool. Users do not need to manually write YAML files or remember cumbersome kubectl commands, greatly reducing the operational threshold for deployment.

[0066] Step 106: If deployment is detected as successful, a performance test is performed to obtain the current performance metrics, and the current performance metrics are compared with the historical best performance metrics corresponding to the hardware resource information in the historical deployment database.

[0067] The successful startup of the container and the start of the model do not mean the deployment is complete. Whether the model can run is one thing, and how well it runs is another. The purpose of this step is to perform a "check-up" on the newly deployed model service to see what kind of performance it can actually achieve under the current environment and parameters.

[0068] The deployment tool continuously monitors the container's running status. Once it confirms that the container has started successfully and the service is responding normally, it will automatically trigger a performance test. This test typically simulates a certain number of concurrent requests for a period of time, allowing the model service to run under high load, and then records key performance metrics.

[0069] The specific methods for performance testing can be flexibly configured according to the model type and business scenario. For example, the performance testing includes concurrent stress testing, and the current performance indicators include at least one of requests per second, average response latency, and success rate; the historical best performance indicator is the highest number of requests per second or the lowest response latency achieved in a historical deployment under the same hardware resource information.

[0070] For large language models, a common testing method is to prepare a batch of test requests and use tools to send hundreds or even thousands of requests to the model service simultaneously to simulate the access pressure of real users. During the test, metrics such as the number of requests processed per second (usually called throughput), the average response time per request, and the success rate of requests are collected. These metrics can objectively reflect the model's operating efficiency under the current deployment configuration.

[0071] After the test is complete, the deployment tool will obtain a set of current performance metrics. The next step is to find a benchmark for comparison to see how effective this deployment really is. This benchmark comes from the historical deployment database.

[0072] The historical deployment database may store performance records from previous deployments for the same hardware resources (e.g., four A100 graphics cards and 512GB of RAM). Deployment tools will search this database to find the best historical performance metrics achieved with the same or similar hardware configurations. This best historical performance might come from the best results achieved when deploying the same model previously, or it might come from the performance of deploying other models of similar scale.

[0073] After finding the historical best performance metric, the deployment tool compares it with the currently measured metric and calculates a deviation value. For example, if the historical best throughput was 500 requests per second, and the current measured throughput is 480, then the deviation is 4%. This deviation value provides a basis for subsequent decisions: if the deviation is small, it means that the deployment effect is close to the historical best; if the deviation is large, it means that the model may not be fully utilizing the hardware performance under the current parameters, and further optimization is needed.

[0074] Step 107: If the deviation of the comparison result exceeds the preset threshold or a deployment failure is detected, the deployment log is input into the reinforcement learning model; the reinforcement learning model evaluates the deployment according to the preset reward function and outputs the parameter adjustment direction, triggering the large language model to update the deployment parameters according to the parameter adjustment direction, until the deviation of the comparison result does not exceed the preset threshold or the preset number of retries is reached.

[0075] This step is crucial for achieving closed-loop optimization in the entire methodology. Two scenarios can trigger this process: one is that the deployment fails entirely, either by failing to start the container or crashing during runtime; the other is that the deployment succeeds, but the performance test results deviate significantly from the historical best benchmark, indicating that the model is not fully utilizing the hardware's capabilities.

[0076] In either case, the deployment tool collects the logs generated during the deployment process. If the deployment fails, it collects error logs from the container runtime, such as errors indicating insufficient CUDA memory; if performance is not up to standard, it collects records from the performance testing process, including measured throughput, latency, and other data. These logs often contain clues to the problem—why it failed, why it's not running fast enough.

[0077] The collected deployment logs are fed into a reinforcement learning model. This model is pre-designed and has its own "value judgment standard," or reward function. The reward function defines what constitutes a good outcome and what constitutes a bad outcome. For example, successful deployment earns positive points, and successful deployment with fast execution earns additional points; failed deployment deducts points, and wasting resources incurs additional deductions. The reinforcement learning model evaluates the deployment based on the deployment logs and the current situation, combined with the reward function, to determine where the problem lies, and then provides directional suggestions for parameter adjustments.

[0078] This directional suggestion isn't a direct change to specific parameters, but rather an adjustment strategy, such as "the memory utilization is set too high and needs to be lowered" or "the parallelism might be insufficient; we could try adding another GPU." This suggestion will be passed on to the previously invoked large language model.

[0079] Once the large language model receives this adjustment direction, it will combine its original understanding of the model documentation and hardware environment with the circumstances of this failure or underperformance to generate a new set of deployment parameters. This new set of parameters will be more conservative or more aggressive than the previous one, depending on the direction given by the reinforcement learning model.

[0080] After generating the new parameters, the deployment tool returns to step 104 with these parameters, and performs verification, deployment, testing, and comparison again using the lightweight simulator. This process is repeated cyclically, with each cycle representing an attempt. Each failure or performance shortfall provides new experience for the reinforcement learning model. The cycle continues until one of two conditions is met: either the deviation in the performance comparison results falls below a preset threshold, indicating that the deployment was effective enough; or the number of attempts has been exhausted, reaching the preset retry limit. At this point, the system stops attempting and may report a failure or issue a warning to the user.

[0081] Through this mechanism, a failed deployment or unsatisfactory performance does not simply end the process, but rather becomes a learning opportunity. The reinforcement learning model accumulates experience from each attempt, and the large language model adjusts its approach based on this experience. Together, they allow the deployment parameters to gradually approach the optimal solution through iterative iterations.

[0082] This application provides a model deployment method, apparatus, electronic device, and storage medium. After receiving a deployment instruction, the method acquires the document information, hardware resource information, and deployment framework information of the model to be deployed, and performs multimodal feature fusion by combining it with the running data in the historical deployment database. It automatically assembles dynamic prompt words to generate deployment parameters, and then performs containerized deployment after verification by a lightweight simulator. After successful deployment, performance testing is performed and compared with historical best performance indicators. If the performance does not meet the standards or the deployment fails, the deployment log is input into the reinforcement learning model for evaluation. The model's output parameters are adjusted to adjust the direction and trigger the language model to update its parameters until the performance meets the standards or the number of retries is reached.

[0083] Based on the above solution, this application integrates document understanding, parameter generation, resource verification, performance evaluation, and closed-loop optimization into a single system, achieving intelligent model deployment throughout the entire process. The large language model automatically generates initial parameters adapted to the current hardware environment and deployment framework through understanding and fusing multimodal information, eliminating the need for manual document reading and configuration, significantly reducing the deployment threshold and error probability. The lightweight simulator pre-verifies parameters before deployment, avoiding deployment failures due to inappropriate parameters. If performance is substandard or deployment fails after deployment, logs are automatically captured and evaluated using a reinforcement learning model. The reinforcement learning model outputs adjustments, and the large language model updates parameters and retryes, forming a self-healing closed loop, significantly improving fault response speed and deployment success rate. Simultaneously, experience data from each deployment is stored in a historical database for reference in subsequent deployments and model optimization, enabling this application embodiment to continuously learn from history and become increasingly accurate with use.

[0084] Compared with existing technologies that rely on manual experience and static configuration, and lack dynamic adjustment and historical reference, the embodiments of this application effectively solve the problems of low deployment efficiency, slow failure response, and difficulty in continuous performance optimization.

[0085] In one feasible implementation, the method further includes: The hardware resource information, deployment parameters, performance indicators, and success / failure status of this deployment are stored in the historical deployment database.

[0086] Regardless of whether this deployment is ultimately successful or not, the data generated throughout the process represents valuable experience that should be recorded for future reference. The purpose of this step is to save the complete details of this deployment to the historical deployment database, allowing the experience in the historical deployment database to be continuously enriched.

[0087] Specifically, the information that needs to be saved includes several aspects. First is hardware resource information, which refers to the target computing environment used in this deployment, such as the model and quantity of graphics cards, video memory size, RAM capacity, and number of CPU cores. This information is crucial for subsequent queries, as the same parameters can behave completely differently on different hardware.

[0088] Secondly, there are the deployment parameters, which are the parameters ultimately used in this deployment, such as the context length, tensor parallelism, and memory utilization threshold. If multiple retries were performed, the set of parameters that ultimately succeeded is particularly worth recording.

[0089] Thirdly, performance metrics are crucial. If deployment is successful and performance testing is conducted, the measured throughput, response latency, success rate, and other metrics need to be saved. These metrics serve as the basis for judging the effectiveness of the deployment and as benchmarks for future comparisons.

[0090] Finally, the outcome of this deployment is whether it succeeded or failed. If it failed, it may be necessary to also save the reasons for the failure or related error logs for future analysis of which parameters are prone to causing which types of failures.

[0091] Storing this data in a historical deployment database enriches the database. The next time the same model or a model with a similar architecture is deployed, more reference cases can be found in this database. For example, it can show which parameter combinations achieved the best performance under the same hardware configuration; it can also show which parameter combinations are prone to failure, allowing for early avoidance.

[0092] Over time, the historical deployment database accumulates more and more cases, acting like a constantly growing library of experience. Deployment tools can refer to increasingly rich historical data when generating initial parameters, and reinforcement learning models can draw upon more and more cases when evaluating and adjusting parameters. The entire system becomes "smarter" with each deployment, and subsequent deployments are often smoother and more effective than earlier ones.

[0093] In a feasible implementation, the reinforcement learning model employs a proximal policy optimization algorithm, which generates parameters to adjust the direction through a policy network, evaluates the quality of the adjusted direction through a value network, and limits the magnitude of each parameter adjustment.

[0094] For example, the state space of the reinforcement learning model includes the hardware resource information, the current deployment parameters, the vector embedding form of the deployment log, and successful or failed cases in the historical deployment database; the reward function is used to assign a positive reward when the deployment is successful, and to add an additional reward based on the ratio of the actual throughput to the target throughput; to assign a negative reward when the deployment fails, and to add an additional penalty when the resource utilization is lower than a preset threshold.

[0095] To ensure that reinforcement learning models function effectively, a feasible implementation plan can utilize the proximal policy optimization algorithm (often abbreviated as PPO) to construct the model. This algorithm is widely used in reinforcement learning and is characterized by a relatively stable learning process, making it less prone to model behavior malfunction due to excessively large adjustments.

[0096] Within this algorithmic framework, the reinforcement learning model comprises two core components. One is the policy network, whose responsibility is to "make suggestions"—providing recommendations on which parameters should be adjusted based on the current situation. For example, if it sees that GPU memory is running low, it might suggest lowering the memory utilization threshold. The other is the value network, whose responsibility is to "score"—evaluating whether the adjustment suggested by the policy network is effective and how much benefit it is expected to bring. The two networks work together, one responsible for decision-making and the other for evaluation, to jointly optimize and adjust the parameters.

[0097] Another important characteristic of near-end policy optimization algorithms is that they limit the magnitude of each parameter adjustment. In other words, they don't allow parameters to suddenly become too large or too small, but rather ensure that each adjustment is within a relatively safe range. This prevents a single aggressive adjustment from causing complete deployment failure, making the entire optimization process smoother and more controllable.

[0098] To enable reinforcement learning models to make reasonable decisions, they need to be provided with sufficient information, known as their "state space." In a feasible implementation, this state space can include several aspects: First, hardware resource information, such as the number of graphics cards and the amount of video memory in the current environment, which is the basis for determining whether the parameters are feasible. Second, the current deployment parameters, that is, the specific values ​​of the set of parameters being evaluated. Third, the vector embedding form of the deployment logs. The logs themselves are text, which needs to be converted into a numerical form that the reinforcement learning model can understand, so that the reasons for failure and performance bottlenecks can be "understood" by the model. Fourth, successful or failed cases in the historical deployment database. These cases are like summaries of past experience, telling the model what adjustments are effective and what adjustments are prone to problems in similar situations.

[0099] Having the state information, a clear "good or bad standard" is needed to guide the model's learning; this is the reward function. In a feasible implementation, the reward function can be designed as follows: if deployment is successful, a positive reward is given as a base score. If the actual measured throughput is higher than the expected target throughput, additional rewards can be given proportionally to encourage the model to pursue better performance. If deployment fails, a negative reward is given as a penalty. If the failure is accompanied by low resource utilization and waste, additional penalties can be added to remind the model to avoid similar problems in the future.

[0100] With this design, the reinforcement learning model has a clear learning objective—to pursue success and high performance, and to avoid failure and waste. Through repeated trials and feedback, it gradually learns which parameter adjustments are more likely to bring good results, thus making the entire deployment and optimization process increasingly intelligent.

[0101] In a feasible implementation plan, such as Figure 2 As shown, the automatic assembly of the dynamic prompt words is accomplished through the following steps: Step 201: The model parameter quantity and quantization support information in the document information, the graphics card model, number of graphics cards, video memory size, memory size, number of central processing unit cores in the hardware resource information, and the successful deployment parameter cases in the historical running data are structurally spliced ​​together according to the preset prompt word template.

[0102] To enable the large language model to accurately understand the current deployment task and provide reasonable parameter suggestions, the assembly of dynamic prompts needs to follow a certain method, rather than simply listing a bunch of information together. In a feasible implementation plan, the collected information can be structured and pieced together according to a preset template to form a clear and complete "requirements specification".

[0103] Specifically, when assembling dynamic prompts, key content is extracted from previously collected information. From the document information, the number of model parameters is extracted, such as whether it's a 70 billion parameter model or a 13 billion parameter model. The number of parameters directly determines the model's basic requirements for GPU memory and computing power. The model's support for quantization is also extracted, such as whether it supports AWQ, GPTQ, and other quantization methods. Quantization allows the model to use less GPU memory and run faster, but it requires cooperation from the framework and hardware.

[0104] From the hardware resource information, we can extract the graphics card model, such as A100 or V100, as different models have different computing capabilities; the number of graphics cards, whether it is a single card or multiple cards in parallel; the video memory size of each graphics card, which directly determines the size of the model that can be stored; and the memory size and the number of CPU cores, which also affect the efficiency of data preprocessing and postprocessing.

[0105] By extracting historical operational data, we can identify valuable case studies of successful deployment parameters. For example, under similar hardware configurations, we can see what parameter combinations were used and what results were achieved when deploying other models of the same scale. These successful cases serve as valuable lessons, providing a framework for large language models to follow when generating parameters.

[0106] After extracting this key information, it is assembled according to a pre-designed prompt template. This template may be pre-designed, containing fixed prompts and placeholders. For example, the template might begin with "You are now a deployment expert. Please generate deployment parameters for the model based on the following information," followed by a list of "Model parameter count: xxx," "Hardware configuration: xxx graphics cards, each with xxx memory," "Reference successful cases: xxx," etc. The extracted information is then filled into the corresponding positions, forming a complete and logically clear prompt.

[0107] Through this structured assembly method, the large language model obtains a well-organized task description, rather than a jumble of disorganized data. It can clearly understand the size of the model to be deployed, the hardware conditions, and any historical lessons that can be learned, thus providing more accurate deployment parameters suitable for the current scenario.

[0108] Based on the same technical concept, embodiments of this application also provide a model deployment apparatus, such as... Figure 3 As shown, the device includes: The receiving module 301 is used to receive instructions from the user to deploy the model to be deployed to the target computing environment.

[0109] The acquisition module 302 is used to acquire the document information of the model to be deployed, the hardware resource information of the target computing environment, and the preset deployment framework information; and to acquire historical running data in the historical deployment database; the historical running data includes the environment information, deployment parameters, performance indicators, and success / failure status of the model to be deployed in the past.

[0110] The prompt module 303 is used to perform multimodal feature fusion of the document information, the hardware resource information, the deployment framework information and the historical running data, automatically assemble dynamic prompt words based on the fused data, and call a large language model to generate deployment parameters according to the dynamic prompt words.

[0111] The verification module 304 is used to verify the deployment parameters through a lightweight simulator to determine whether the deployment parameters meet the deployment conditions of the model to be deployed.

[0112] The deployment module 305 is used to generate a deployment file based on the deployment parameters and call the target program to perform containerized deployment if the verification is successful.

[0113] The testing module 306 is used to perform performance testing if successful deployment is detected, to obtain the current performance metrics, and to compare the current performance metrics with the historical best performance metrics corresponding to the hardware resource information in the historical deployment database.

[0114] The adjustment module 307 is used to input the deployment log into the reinforcement learning model if the deviation of the comparison result exceeds a preset threshold or a deployment failure is detected. The reinforcement learning model evaluates the deployment according to the preset reward function and outputs the parameter adjustment direction, triggering the large language model to update the deployment parameters according to the parameter adjustment direction, until the deviation of the comparison result does not exceed the preset threshold or the preset number of retries is reached.

[0115] In one feasible implementation, the device further includes: The storage module is used to store the hardware resource information, deployment parameters, performance indicators, and success / failure status of this deployment into the historical deployment database.

[0116] In a feasible implementation, the reinforcement learning model employs a proximal policy optimization algorithm, which generates parameters to adjust the direction through a policy network, evaluates the quality of the adjusted direction through a value network, and limits the magnitude of each parameter adjustment.

[0117] In one feasible implementation, the state space of the reinforcement learning model includes the hardware resource information, current deployment parameters, vector embeddings of deployment logs, and success or failure cases in the historical deployment database.

[0118] The reward function is used to assign a positive reward when deployment is successful, and to add an additional reward based on the ratio of actual throughput to target throughput; to assign a negative reward when deployment fails, and to add an additional penalty when resource utilization is lower than a preset threshold.

[0119] In one feasible implementation, the lightweight simulator is a pre-built video memory calculation algorithm used to estimate video memory usage and determine whether the deployment parameters will cause video memory overflow based on the number of parameters of the model to be deployed, the number of graphics cards, and the size of video memory per card.

[0120] In one feasible implementation, the performance test includes concurrent stress testing, and the current performance metrics include at least one of requests processed per second, average response latency, and success rate.

[0121] The historical best performance metric refers to the highest number of requests per second or the lowest response latency achieved in a historical deployment under the same hardware resource information.

[0122] In one feasible implementation, the automatic assembly of the dynamic prompt words is accomplished through the following steps: The model parameter quantity and quantization support information in the document information, the graphics card model, number of graphics cards, video memory size, memory size, number of central processing unit cores in the hardware resource information, and the successful deployment parameter cases in the historical running data are structurally spliced ​​together according to the preset prompt word template.

[0123] Figure 4 A schematic diagram of an electronic device provided in this application embodiment includes: a processor 401, a storage medium 402, and a bus 403. The storage medium 402 stores machine-readable instructions executable by the processor 401. When the electronic device runs the model deployment method as described in the embodiment, the processor 401 communicates with the storage medium 402 via the bus 403, and the processor 401 executes the machine-readable instructions to perform the steps as described in the embodiment.

[0124] In this embodiment, the storage medium 402 may also execute other machine-readable instructions to perform other methods as described in the embodiment. For details on the specific execution steps and principles, please refer to the description of the embodiment, which will not be repeated here.

[0125] This application also provides a computer-readable storage medium storing a computer program that is executed by a processor to perform the steps as described in the embodiments.

[0126] In this embodiment, the computer program, when run by the processor, can also execute other machine-readable instructions to perform other methods as described in the embodiments. For details on the specific execution steps and principles, please refer to the description of the embodiments, which will not be repeated here.

[0127] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the coupling or direct coupling or communication connection shown or discussed may be through some communication interface; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0128] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0129] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0130] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0131] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A model deployment method, characterized in that, The method includes: Receive instructions from users to deploy the model to the target computing environment; The system acquires the document information of the model to be deployed, the hardware resource information of the target computing environment, and the preset deployment framework information; and acquires historical running data from the historical deployment database; the historical running data includes the environment information, deployment parameters, performance indicators, and success / failure status of the model to be deployed in the past. The document information, hardware resource information, deployment framework information, and historical operation data are fused using multimodal features. Based on the fused data, dynamic prompt words are automatically assembled, and a large language model is invoked to generate deployment parameters according to the dynamic prompt words. The deployment parameters are verified using a lightweight simulator to determine whether they meet the deployment conditions of the model to be deployed. If the verification passes, a deployment file is generated based on the deployment parameters, and the target program is invoked to perform containerized deployment. If deployment is detected as successful, a performance test is performed to obtain the current performance metrics, and the current performance metrics are compared with the historical best performance metrics corresponding to the hardware resource information in the historical deployment database. If the deviation of the comparison result exceeds the preset threshold or a deployment failure is detected, the deployment log is input into the reinforcement learning model. The reinforcement learning model evaluates the deployment according to the preset reward function and outputs the parameter adjustment direction, triggering the large language model to update the deployment parameters according to the parameter adjustment direction, until the deviation of the comparison result does not exceed the preset threshold or the preset number of retries is reached.

2. The method according to claim 1, characterized in that, The method further includes: The hardware resource information, deployment parameters, performance indicators, and success / failure status of this deployment are stored in the historical deployment database.

3. The method according to claim 1, characterized in that, The reinforcement learning model employs a proximal policy optimization algorithm, which generates parameters to adjust the direction through a policy network, evaluates the quality of the adjusted direction through a value network, and limits the magnitude of each parameter adjustment.

4. The method according to claim 3, characterized in that, The state space of the reinforcement learning model includes the hardware resource information, the current deployment parameters, the vector embedding form of the deployment log, and the success or failure cases in the historical deployment database. The reward function is used to assign a positive reward when deployment is successful, and to add an additional reward based on the ratio of actual throughput to target throughput; to assign a negative reward when deployment fails, and to add an additional penalty when resource utilization is lower than a preset threshold.

5. The method according to claim 1, characterized in that, The lightweight simulator has a pre-built video memory calculation algorithm, which is used to estimate video memory usage and determine whether the deployment parameters will cause video memory overflow based on the number of parameters of the model to be deployed, the number of graphics cards, and the size of video memory per card.

6. The method according to claim 1, characterized in that, The performance test includes concurrent stress testing, and the current performance metrics include at least one of the following: requests processed per second, average response latency, and success rate. The historical best performance metric refers to the highest number of requests per second or the lowest response latency achieved in a historical deployment under the same hardware resource information.

7. The method according to claim 1, characterized in that, The automatic assembly of the dynamic prompt words is accomplished through the following steps: The model parameter quantity and quantization support information in the document information, the graphics card model, number of graphics cards, video memory size, memory size, number of central processing unit cores in the hardware resource information, and the successful deployment parameter cases in the historical running data are structurally spliced ​​together according to the preset prompt word template.

8. A model deployment device, characterized in that, The device includes: The receiving module is used to receive instructions from users to deploy the model to the target computing environment. The acquisition module is used to acquire the document information of the model to be deployed, the hardware resource information of the target computing environment, and the preset deployment framework information; and to acquire historical running data from the historical deployment database; the historical running data includes the environment information, deployment parameters, performance indicators, and success / failure status of the model to be deployed in the past. The prompt module is used to perform multimodal feature fusion of the document information, hardware resource information, deployment framework information and historical running data, automatically assemble dynamic prompt words based on the fused data, and call a large language model to generate deployment parameters according to the dynamic prompt words; The verification module is used to verify the deployment parameters through a lightweight simulator to determine whether the deployment parameters meet the deployment conditions of the model to be deployed. The deployment module is used to generate a deployment file based on the deployment parameters and call the target program to perform containerized deployment if the verification passes. The testing module is used to perform performance tests to obtain the current performance metrics if successful deployment is detected, and to compare the current performance metrics with the historical best performance metrics corresponding to the hardware resource information in the historical deployment database. An adjustment module is used to input deployment logs into a reinforcement learning model if the deviation of the comparison result exceeds a preset threshold or a deployment failure is detected. The reinforcement learning model evaluates the deployment based on a preset reward function and outputs a parameter adjustment direction, triggering the large language model to update the deployment parameters according to the parameter adjustment direction, until the deviation of the comparison result does not exceed the preset threshold or the preset number of retries is reached.

9. An electronic device, characterized in that, include: The device includes a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the model deployment method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the model deployment method as described in any one of claims 1 to 7.

Images