Model deployment method, electronic equipment, storage medium and program product

Abstract

The invention discloses a model deployment method, electronic equipment, a storage medium and a program product, and relates to the technical field of computers, hardware environment detection is performed on a server to obtain hardware detection information, and the hardware detection information comprises graphics card information and resource residual information; the video card information comprises server types corresponding to a plurality of video cards of the server and single-card residual video memories of the video cards; through a recommendation model corresponding to the server type, the display card information and the resource residual information are analyzed and processed, a model information list corresponding to at least one to-be-deployed model is obtained, and the model information list comprises a model identifier, model information and deployment configuration information corresponding to each to-be-deployed model; determining a video memory allocation strategy corresponding to the to-be-deployed model according to the hardware detection information and a model information list corresponding to the at least one to-be-deployed model; and based on the video memory allocation strategy, executing deployment of the at least one to-be-deployed model. The reliability of model deployment can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to model deployment methods, electronic devices, storage media, and program products. Background Technology

[0002] With the rapid development of artificial intelligence technology, the demand for private deployment of large-scale models is growing. Enterprise users typically need to deploy large models on local servers or data centers to meet requirements such as data privacy, real-time response, and high availability.

[0003] In related technologies, users need to manually check whether the server has a graphics processing unit (GPU) installed, whether the computing memory capacity meets the model parameter requirements, and install and configure environment dependency packages. However, deploying large models requires manual estimation of memory requirements, which cannot accurately match the memory consumption of model training / inference, resulting in low reliability of model deployment. Summary of the Invention

[0004] This application provides a model deployment method, electronic device, storage medium, and program product to at least address the problem of low order delivery efficiency in related technologies.

[0005] Firstly, this application provides a model deployment method, including:

[0006] The server undergoes hardware environment testing to obtain hardware testing information, which includes graphics card information and remaining resource information. The graphics card information includes the server type corresponding to the multiple graphics cards of the server, as well as the remaining video memory of each graphics card.

[0007] By using the recommendation model corresponding to the server type, the graphics card information and the remaining resource information are analyzed and processed to obtain a model information list corresponding to at least one model to be deployed. The model information list includes the model identifier, model information and deployment configuration information corresponding to each model to be deployed.

[0008] Based on the hardware detection information and the model information list corresponding to the at least one model to be deployed, the video memory allocation strategy corresponding to the model to be deployed is determined;

[0009] Based on the aforementioned memory allocation strategy, the deployment of at least one model to be deployed is executed.

[0010] Secondly, embodiments of this application provide a model deployment apparatus, including a detection module, an analysis module, a determination module, and an execution module:

[0011] The detection module is used to perform hardware environment detection on the server and obtain hardware detection information, which includes graphics card information and remaining resource information. The graphics card information includes the server type corresponding to the multiple graphics cards of the server and the remaining video memory of each graphics card.

[0012] The analysis module is used to analyze and process the graphics card information and the remaining resource information through the recommendation model corresponding to the server type, and obtain a model information list corresponding to at least one model to be deployed. The model information list includes the model identifier, model information and deployment configuration information corresponding to each model to be deployed.

[0013] The determining module is used to determine the video memory allocation strategy corresponding to the model to be deployed based on the hardware detection information and the model information list corresponding to the at least one model to be deployed.

[0014] The execution module is used to deploy the at least one model to be deployed based on the video memory allocation strategy.

[0015] Thirdly, this application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described model deployment methods.

[0016] Fourthly, this application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described model deployment methods.

[0017] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described model deployment methods.

[0018] The model deployment method, electronic device, storage medium, and program product provided in this application embodiment do not require manual estimation of video memory requirements. They achieve accurate calculation of video memory requirements through automated hardware detection and intelligent recommendation models, and dynamically generate video memory allocation strategies by combining multi-dimensional hardware detection information, directly guiding the entire model deployment process. This can significantly reduce the cost of manual intervention and operational errors, and significantly improve the reliability of model deployment. Attached Figure Description

[0019] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A schematic diagram illustrating the application scenarios provided in the embodiments of this application;

[0021] Figure 2 A schematic flowchart illustrating a model deployment method provided in an embodiment of this application;

[0022] Figure 3 A flowchart illustrating another model deployment method provided in an embodiment of this application;

[0023] Figure 4 This is a schematic diagram of the structure of a model deployment device provided in an embodiment of this application;

[0024] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0026] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0027] Figure 1 This is a schematic diagram illustrating an application scenario provided in an embodiment of this application. Please refer to [link / reference]. Figure 1 This application scenario can include user device 101 and server 102 to be deployed. With the rapid development of artificial intelligence technology, the demand for private deployment of large models is increasing. Users can deploy large models on server 102 through user device 101. Server 102 to be deployed can be a local server or a data center server to meet core business requirements such as data privacy protection, low-latency real-time response, and high availability services.

[0028] In related technologies, users need to manually check whether the server has a GPU, whether the compute memory capacity meets the model parameter requirements, and install and configure environment dependency packages. However, deploying large models requires manual estimation of compute memory requirements, which cannot accurately match the compute memory consumption of model training / inference, resulting in low reliability of model deployment.

[0029] The model deployment method provided in this application embodiment performs hardware environment detection on the server to obtain hardware detection information, including graphics card information and remaining resource information. The graphics card information includes the server type corresponding to the multiple graphics cards on the server, and the remaining video memory of each graphics card. The graphics card information and remaining resource information are analyzed and processed using a recommended model corresponding to the server type to obtain a model information list corresponding to at least one model to be deployed. The model information list includes the model identifier and model information corresponding to each model to be deployed. Based on the hardware detection information and the model information list corresponding to at least one model to be deployed, a video memory allocation strategy corresponding to the model to be deployed is determined. Based on the video memory allocation strategy, the deployment of at least one model to be deployed is executed.

[0030] The above execution process does not require manual estimation of video memory requirements. It only needs to achieve accurate calculation of video memory requirements through automated hardware detection and intelligent recommendation models, and generate video memory allocation strategies by combining multi-dimensional hardware detection information. This directly guides the entire model deployment process, which can significantly reduce the cost of manual intervention and operational errors, and improve the reliability of model deployment.

[0031] Figure 2 This is a flowchart illustrating a model deployment method provided in an embodiment of this application. Please refer to [link / reference]. Figure 2 The method may include:

[0032] S201. Perform hardware environment testing on the server and obtain hardware testing information.

[0033] Hardware detection information can include graphics card information and remaining resource information.

[0034] Graphics card information can include the number of graphics cards corresponding to the server's multiple graphics cards, the server type, the video memory of each graphics card, the graphics card model, the remaining video memory of a single card, and the interface type.

[0035] Resource remaining information includes remaining CPU, memory, and storage information.

[0036] S202. Using the recommended model corresponding to the server type, analyze and process the graphics card information and remaining resource information to obtain a list of model information corresponding to at least one model to be deployed.

[0037] The recommended model is a model trained on a historical deployment dataset. Specifically, it can be trained using gradient boosting trees, extreme gradient boosting (XGBoost), and deep learning regression models.

[0038] The historical deployment dataset can include multiple standardized historical deployment data, which consists of historical hardware information, deployment configuration information, and historical model information recorded during past model deployments.

[0039] Historical hardware information can include graphics card type, number of graphics cards, total / remaining video memory per card, graphics card architecture, etc. Deployment configuration information includes model accuracy, batch size, environment dependency version, etc. Historical model information includes model identifier, model architecture (Transformer / GNN, etc.), parameter scale, actual peak video memory usage, whether cross-video memory scheduling is required, etc.

[0040] Multiple historical deployment data are preprocessed data. Data preprocessing may include outlier removal, missing value imputation, feature standardization, feature engineering, and splitting the training set / test set.

[0041] When training the model using historical deployment data, historical hardware information is used as the model input features, and deployment configuration information and historical model information are used as dual output targets. The memory usage prediction accuracy is optimized by minimizing the MSE (mean squared error) loss function, and the classification accuracy for cross-memory scheduling requirements is optimized by using the cross-entropy loss function. During training, the model output is periodically compared with the actual data of the historical model (actual memory usage, whether cross-memory scheduling occurs), and feedback is provided to iteratively adjust model parameters (such as learning rate and tree depth) until the model prediction error is lower than a preset threshold (e.g., memory usage prediction error ≤ 5%), thus completing model training.

[0042] The hardware detection information is the real-time hardware information of the server where the model needs to be deployed, specifically including graphics card information (graphics card type, quantity, total video memory / used video memory / remaining video memory per card, communication bandwidth) and remaining resource information (remaining CPU computing power, remaining memory capacity, disk I / O bandwidth, network transmission rate).

[0043] Inputting the hardware detection information into the trained recommendation model will output the deployment configuration information and model information corresponding to at least one model to be deployed that can be supported by the current server. The deployment configuration information and model information corresponding to at least one model to be deployed can be presented through the model information list.

[0044] Since different servers may have one or more graphics cards installed, and the types of graphics cards may be the same or different, the server's graphics card configuration is first identified through hardware detection information, and then the servers are classified according to "single graphics card type", "multiple graphics cards of the same type" and "multiple graphics cards of different types".

[0045] Among them: Single graphics card type means that the server has only one graphics card deployed; Multiple graphics cards of the same type means that the server has two or more graphics cards deployed and all graphics cards are of the same type; Multiple graphics cards of different types means that the server has two or more graphics cards deployed and there are different types of graphics cards, such as NVIDIA A100 and AMD MI250.

[0046] In some embodiments, hardware detection information can be analyzed and processed based on the recommendation model corresponding to the server type to obtain a compatible recommendation list corresponding to the model to be deployed. The system receives a list of model information corresponding to at least one model to be deployed selected by the user from the compatible recommendation list.

[0047] The compatibility recommendation list not only includes multiple recommendation model identifiers, but also model information and deployment configuration information. The model information includes the number of parameters, estimated peak memory usage, recommended deployment accuracy, estimated deployment time, and resource usage percentage. The deployment training information includes whether cross-memory scheduling is required and the recommended deployment mode (single card / multi-card collaboration).

[0048] The model information list may include key information such as the model identifier corresponding to at least one model to be deployed, deployment configuration information (such as accuracy and batch size) and model information (number of parameters and architecture type) corresponding to each model to be deployed selected by the user.

[0049] The compatibility recommendation list is a set of models that are fully compatible with the server, obtained by combining the recommendation model with real-time hardware detection information.

[0050] The model information list is a list of deployment objects that users select based on their own business needs (such as inference speed and model accuracy), while being compatible with the recommendation list.

[0051] In this application, users can select one or more models to be deployed from a compatible recommendation list, enabling flexible model selection interaction. This ensures that the final model to be deployed fully complies with the current server's hardware environment and resource constraints, and accurately matches the user's actual business needs, thereby significantly improving the flexibility and practicality of model deployment.

[0052] S203. Based on the hardware detection information and the model information list corresponding to at least one model to be deployed, determine the video memory allocation strategy corresponding to the model to be deployed.

[0053] The video memory allocation strategy is a core solution used to clarify the video memory allocation method, graphics card mapping relationship, and resource scheduling rules for the model to be deployed on the current server. Specifically, it includes: the binding relationship between the model and the graphics card, single / multi-card deployment mode, video memory usage threshold, dynamic scheduling mechanisms (such as time-sharing and video memory sharing), and the selection of cross-graphics card data transmission protocols (NVLink / PCIe). PCIe stands for Peripheral Component Interconnect Express. NVLink is NVIDIA's proprietary high-speed interconnect technology (NVIDIA Link).

[0054] Whether there is a single model or multiple models to be deployed, server resources must be allocated based on hardware resource constraints (video memory, computing power, communication bandwidth). For a single model, the focus should be on ensuring deployment stability and optimal performance, while for multiple models, resource utilization and priority adaptation should be taken into account.

[0055] In some embodiments, the estimated peak video memory usage of each model to be deployed in at least one module is determined based on the model information list corresponding to the model to be deployed. If there is only one model to be deployed, the video memory allocation strategy is determined based on the estimated peak video memory usage of the model to be deployed and the remaining video memory of each graphics card. If there are multiple models to be deployed, the video memory allocation strategy is determined based on the priority of each model to be deployed, the estimated peak video memory usage, and the remaining video memory of each graphics card.

[0056] The estimated peak memory usage can be the maximum memory consumption of the model to be deployed during the entire process of weight loading, inference calculation, intermediate result caching, etc., which is the peak memory usage.

[0057] In this application, by designing a memory allocation strategy based on the number of models to be deployed and the model priority, we can ensure optimal performance for single-model deployment and achieve efficient resource utilization for multi-model deployment. This not only improves deployment efficiency but also significantly enhances deployment reliability and stability.

[0058] S204. Based on the video memory allocation strategy, execute the deployment of at least one model to be deployed.

[0059] In some embodiments, preset compatibility rule information is obtained; if there is only one model to be deployed, the dependency packages of each model to be deployed are installed based on the preset compatibility rule information, and the model configuration file corresponding to the model to be deployed is obtained from the dependency package. The model configuration file is configured according to the memory allocation strategy; if there are multiple models to be deployed, the conflict of the dependency package versions corresponding to each model to be deployed is checked and the conflict check result is obtained; the multiple models to be deployed are deployed according to the memory allocation strategy, the preset compatibility rule information and the conflict check result.

[0060] The default compatibility rules information is a pre-written, version-controlled, standardized non-markup language (YAML Ain'tMarkup Language) configuration file. This information records the version compatibility relationships of basic dependencies, the adaptation rules of frameworks and models, the dependency conflict determination rules, the optimized dependency recommendation list, and the offline resource mapping relationships.

[0061] The dependency packages are the core software components that support the operation of the model to be deployed, including: underlying driver dependencies, deep learning framework dependencies, core algorithm library dependencies, service deployment dependencies, optimization tool dependencies, and other third-party auxiliary dependencies. Each dependency package is pre-stored in the local offline repository on the server, supporting version tracking and quick invocation.

[0062] This application enables automated, unmanned deployment of models (single or multiple) to be deployed, which solves the problem of multi-model dependency conflicts and avoids the impact of the network environment on deployment. At the same time, by binding the memory allocation strategy with the deployment process, it ensures the precise matching of model deployment with hardware resources, which greatly improves the efficiency, stability and reliability of deployment.

[0063] The model deployment method provided in this application embodiment can eliminate the need for manual estimation of video memory requirements. It can accurately calculate video memory requirements through automated hardware detection and intelligent recommendation models, and dynamically generate video memory allocation strategies by combining multi-dimensional hardware detection information to directly guide the entire model deployment process. This can significantly reduce the cost of manual intervention and operational errors, while solving the compatibility problem of multi-GPU collaborative deployment and the problem of video memory resource waste in related technologies, and can significantly improve the reliability of model deployment.

[0064] Figure 3 This is a flowchart illustrating another model deployment method provided in an embodiment of this application. Please refer to... Figure 3 The method may include:

[0065] S301. Perform hardware environment testing on the server and obtain hardware testing information.

[0066] The execution process of S301 can be found in the execution process of S201, and will not be repeated here.

[0067] S302. Based on the recommended model corresponding to the server type, analyze and process the hardware detection information to obtain a compatible recommendation list corresponding to the model to be deployed.

[0068] The compatibility recommendation list can include multiple recommendation model identifiers, model information corresponding to each recommendation model identifier, and deployment configuration information.

[0069] In some embodiments, if the server type is a multi-graphics card of different types, the recommendation model is a multi-graphics card recommendation model. The hardware detection information is analyzed and processed through the multi-graphics card recommendation model to obtain a compatible recommendation list. If the server type is a single-graphics card type or a multi-graphics card of the same type, the recommendation model is a single-graphics card recommendation model. The hardware detection information is analyzed and processed through the single-graphics card recommendation model to obtain a compatible recommendation list.

[0070] The multi-graphics card recommendation model is trained based on historical heterogeneous deployment data (including data on the compatibility, performance loss, and cross-graphics card communication efficiency of different graphics card combinations and model architectures).

[0071] When training recommendation models for multiple types of graphics cards, we focus on combining graphics card type compatibility, communication bandwidth (NVLink / PCIe version), and heterogeneous collaboration support capabilities from hardware detection information to select models that can be deployed in a distributed manner using multiple types of graphics cards and whose performance loss is lower than a preset threshold (e.g., ≤15%).

[0072] Meanwhile, the compatibility recommendation list clearly indicates the optimal graphics card combination for heterogeneous deployment and the cross-graphics card data transfer strategy. For example, the optimal graphics card combination for model X is a collaborative deployment of 2×A100 + 1×MI250.

[0073] The single-type graphics card recommendation model is trained based on historical deployment data of graphics cards of the same type.

[0074] When training a model for a single type of graphics card, the focus is on combining the remaining video memory of a single card, the number of graphics cards, and the communication bandwidth of the same card in the hardware detection information to select models that match the video memory requirements with hardware resources and support single-card / multi-card parallelism (data parallelism / model parallelism).

[0075] The single-type graphics card recommendation model can calculate the maximum model parameter scale that the current hardware can support in real time based on the available video memory and quantity of a single card. For example, a graphics card with 16GB of video memory can be automatically determined to support a model with 7 billion parameters.

[0076] In particular, for scenarios with multiple graphics cards of the same type, the recommended parallel splitting method should also be indicated; for example, model Y is recommended to have 4 cards for data parallel processing, with a batch size of 16.

[0077] After receiving hardware detection information (graphics card information and remaining resource information), the recommendation model first filters out a set of models that are basically compatible with the hardware through feature matching (e.g., memory requirement ≤ total available memory × 1.1). Then, it sorts the models in the set through the model's built-in fit scoring algorithm, and finally selects the top N models to generate a compatible recommendation list, ensuring that the recommendation results meet both hardware constraints and the user's potential performance needs.

[0078] When filtering the model set, the models in the model set can meet the requirement that the video memory requirement is less than or equal to a preset multiple of the total available video memory in the hardware, where the preset multiple can be 1.1.

[0079] N is an integer greater than 1. N is configurable; for example, N=10.

[0080] Among them, the fit scoring algorithm can comprehensively consider factors such as memory utilization, inference performance, and deployment complexity.

[0081] In this application, by matching a dedicated and optimized recommendation model according to server type, the prediction bias of the generalized model in different scenarios such as heterogeneous multi-card and homogeneous multi-card is avoided. At the same time, it is compatible with the multi-dimensional model information and deployment adaptation data attached to the recommendation list, which solves the problem of model recommendation and hardware scenario mismatch in related technologies. Furthermore, the precise filtering and sorting reduces the user's selection cost, thereby significantly improving the adaptability, efficiency and final operational reliability of model deployment.

[0082] S303. Receive a list of model information corresponding to at least one model to be deployed selected by the user from the compatibility recommendation list.

[0083] A compatible recommendation list can be displayed through a front-end visual interactive interface, allowing users to narrow down their selections by filtering criteria such as model parameter count, inference latency, and deployment mode.

[0084] It can provide two operation modes: "one-click selection of the optimal model" (the system automatically recommends models based on the fit score) and "custom multi-selection". It can receive at least one model to be deployed from the compatibility recommendation list and then generate a structured list of model information.

[0085] In addition to the identifiers of each model to be deployed selected by the user, the model information list should also automatically supplement the following key information: the deployment configuration confirmed by the user, the model deployment priority set by the user, the service startup parameter requirements, and the retry strategy after deployment failure (such as the number of retries and the retry interval), to ensure that the list information fully covers the requirements of the entire process of subsequent memory allocation, dependency installation, and service startup.

[0086] User-confirmed deployment configurations may include inference accuracy, batch size, and whether quantization optimization is enabled; user-set model deployment priorities can be dragged and dropped for sorting, and are sorted by the recommended list's suitability by default; service startup parameters may include specifying the port number, log storage path, and whether to enable the Application Programming Interface (API) service.

[0087] During the interaction, the system can provide a real-time feedback mechanism: when a user selects a single model, the interface displays the model's final estimated resource usage (GPU memory / CPU / memory), inference performance metrics (throughput, response latency), and deployment time in real time.

[0088] When a user selects multiple models, the interface automatically checks whether the total resource requirements of the selected models exceed the hardware's available resource threshold (e.g., total video memory requirements ≤ total available video memory × 1.2). If it does, a pop-up window will display "Insufficient resources" and recommend removing low-fitness models or adjusting the deployment configuration (e.g., reducing the accuracy of some models). At the same time, users can reselect to avoid subsequent deployment failures due to user misselection.

[0089] In addition, the system supports users to edit the selected model: it allows modification of configurable parameters such as deployment accuracy and batch size (with real-time updates of estimated resources and performance data after modification), and also allows the addition of custom tags (such as "core business model" and "test model") for easy subsequent management.

[0090] After the user confirms the selection, the system automatically stores the list of model information in a standardized JavaScript Object Notation (JSON) format in the local configuration directory and generates a unique task ID associated with the deployment task, supporting deployment process traceability and subsequent task management (such as pausing, restarting, and canceling).

[0091] In this application, through visual interaction and real-time verification mechanisms, users can not only clearly understand the deployment feasibility and expected effects of each model, but also provide a clear basis for subsequent memory allocation strategy formulation and dependent environment configuration. This not only ensures the flexibility and autonomy of user choices, but also avoids deployment risks caused by user operation errors through resource pre-verification, parameter standardization and other mechanisms, further strengthening the stability and closed-loop nature of the deployment process.

[0092] S304. Based on the model information list corresponding to the model to be deployed, determine the estimated peak memory usage of each model to be deployed in at least one module to be deployed.

[0093] The estimated peak memory usage can be the peak memory usage from the prediction results of the recommendation model.

[0094] In some embodiments, dynamic calibration can be further combined with the following multi-dimensional factors: user-confirmed deployment configuration parameters, model architecture and weight characteristics, hardware and framework adaptation effects, etc.

[0095] The estimated value can be adjusted according to the inference precision selected by the user, based on the precision and memory usage conversion formula.

[0096] For different architecture models such as Transformer, Graph Neural Network (GNN), and Convolutional Neural Network (CNN), the pre-defined architecture-specific GPU memory calculation factor in the recommendation model is called. For example, the Transformer model needs to reserve additional GPU memory for the intermediate results cache of the self-attention mechanism, which accounts for about 20%-30% of the weight GPU memory.

[0097] By combining the graphics card type, graphics card driver version, and deep learning framework version from the hardware detection information, the estimated value is adjusted for compatibility to ensure consistency with the actual deployment environment.

[0098] You can add 10%-15% redundant video memory reservation to the estimated peak video memory usage of each model to be deployed (the ratio can be adjusted through the configuration file) to deal with instantaneous video memory fluctuations during the deployment process (such as weight decompression when loading the model and temporary data caching during inference).

[0099] The estimated peak memory usage, calibration criteria (such as accuracy, batch size, optimization strategy), and redundancy ratio of each model are recorded in the model information list and can be synchronously fed back to the front-end interface for users to view, providing a reference for users to make secondary adjustments to the deployment configuration.

[0100] S305. If there is only one model to be deployed, the memory allocation strategy shall be determined based on the estimated peak memory usage of the model to be deployed and the remaining memory of each graphics card.

[0101] Specifically, the difference between the remaining video memory of each card and the estimated peak video memory usage is calculated to obtain the cache space difference for each graphics card; it is then determined whether there is a graphics card whose cache space difference is greater than or equal to the first cache threshold; if so, the video memory allocation strategy is a single-card independent video memory strategy; if not, the video memory allocation strategy is a multi-card collaborative deployment strategy.

[0102] First, the single-card independent video memory strategy means that a single graphics card provides video memory for the model to be deployed.

[0103] If there is a target graphics card with a cache space difference greater than or equal to the first cache threshold, the model is bound to that target graphics card. At the same time, the upper limit of video memory usage is set to the estimated peak video memory usage. This can avoid occupying redundant video memory and affecting other potential tasks. The graphics card is configured with exclusive permissions. The target graphics card ID can be locked through the CUDA_VISIBLE_DEVICES environment variable.

[0104] CUDA stands for Compute Unified Device Architecture.

[0105] If multiple graphics cards meet the requirements, the optimal graphics card is selected as the target graphics card based on its computing power to ensure the model's performance.

[0106] Secondly, if there are no graphics cards with a cache space difference greater than or equal to the first cache threshold, a multi-card collaborative deployment strategy can be adopted, as detailed below.

[0107] If the server type is a multi-GPU system of the same type: prioritize multi-GPU model parallelism / data parallelism strategy.

[0108] Based on the model architecture, the model is split into multiple graphics cards according to the proportion of video memory required. For example, if the model estimates a peak video memory usage of 16GB, and each of the two graphics cards of the same type has 10GB remaining, it can be split into "8GB + 8GB" and a data synchronization mechanism can be configured (such as using full synchronization mode when NVLink bandwidth is ≥300GB / s, and optimizing the synchronization frequency when PCIe 4.0 bandwidth is used).

[0109] If the server type is multi-GPU heterogeneous: when the recommendation model determines that "the model supports heterogeneous collaboration", a heterogeneous multi-GPU collaborative deployment strategy is adopted.

[0110] Computationally intensive modules (such as Transformer encoders / decoders) can be prioritized for allocation to graphics cards with stronger computing power (such as A100), while memory-intensive modules (such as weight storage and intermediate result caching) can be allocated to graphics cards with larger video memory (such as MI250). Low-latency data transmission across graphics cards can be achieved through the PCIe / NVLink protocol. At the same time, a heterogeneous collaborative adaptation layer can be set up to avoid compatibility issues caused by type differences.

[0111] If the server is a single-graphics card type (only 1 graphics card and the difference in cache space is less than the first cache threshold): trigger a deployment warning, the front-end pop-up window prompts the user "single card video memory is insufficient", and recommends solutions (such as reducing model inference accuracy, reducing batch size, enabling quantization optimization, and simultaneously displaying the adjusted estimated peak video memory usage). After the user confirms the adjustment, repeat this step or terminate the deployment process.

[0112] For example, assuming the model to be deployed is a 13B parameter Transformer generation model, the estimated peak memory usage (including redundancy) at half-precision floating point (FP16) precision is 14GB; the server is a multi-GPU of the same type (2 NVIDIA A10G GPUs, each with 12GB of remaining memory): the cache space difference 1 is calculated as ΔG1 = 12GB - 14GB = -2GB, and the cache space difference 2 is calculated as ΔG2 = 12GB - 14GB = -2GB, both of which are less than the first cache threshold of 0. Therefore, a multi-GPU model parallel strategy is adopted, splitting the model into two parts by layer, each occupying 8GB of memory (the remaining memory is used for data synchronization). Cross-GPU data transfer is achieved through NVLink to ensure that the model runs normally and the inference latency is controllable.

[0113] This application combines multiple factors such as model architecture, graphics card type, and communication bandwidth to avoid the limitations of simple difference-based judgments or equal-division strategies. Simultaneously, it provides dynamic adjustment suggestions for scenarios with insufficient single-card memory, achieving precise adaptation and optimal performance for single-model deployment in different hardware environments. This refined strategy improves resource utilization and deployment success rate, thereby enhancing the reliability, flexibility, and operational performance of model deployment.

[0114] S306. If there are multiple models to be deployed, the video memory allocation strategy shall be determined based on the priority of each model to be deployed, the estimated peak video memory usage, and the remaining video memory of each graphics card.

[0115] In some embodiments, a model priority queue corresponding to multiple models to be deployed can be determined, and the model priority queue is arranged from high to low priority according to the priority of each model to be deployed; the sum of the remaining video memory of each graphics card is determined as the total remaining video memory of multiple graphics cards; the total video memory usage of multiple models to be deployed is determined according to the estimated peak video memory usage of each model to be deployed; the total cache difference between the total remaining video memory and the total video memory usage is determined; it is determined whether the total cache difference is greater than or equal to a second cache threshold; if so, the model priority queue is divided into front-end models to be deployed and tail-end models to be deployed; the video memory allocation strategy of the front-end models to be deployed is determined to be a single-card independent video memory strategy; the video memory allocation strategy of the tail-end models to be deployed is determined to be a video memory time-sharing strategy; if not, the video memory of the tail-end models to be deployed in the model priority queue is compressed to ensure that the total cache difference is greater than or equal to the second cache threshold.

[0116] When building a model priority queue, the model priority queue can be built based on the priority of each model to be deployed (such as core business model > regular model > test model, and the priority weight can be configured). The queue is arranged from high to low priority. If there are models with the same priority, they are sorted a second time according to the estimated peak memory usage from small to large or the deployment timeliness requirements from high to low, so as to ensure that the queue sorting takes into account multiple needs.

[0117] The following is the formula for calculating the total cache difference provided in this application:

[0118]

[0119] in, To cache the total difference, This represents the total remaining video memory. This represents the total video memory usage.

[0120] The second cache threshold can be a preset value, and it can be used to determine whether the server's total resources are sufficient. The threshold can be set proportionally to the total video memory of the graphics cards. For example, the second cache threshold could be 60% of the total video memory.

[0121] If the total cache difference ΔTotal is greater than or equal to the second cache threshold, it means the server has sufficient total resources:

[0122] Based on the model priority queue order and the condition of "remaining video memory on a single card ≥ estimated peak video memory usage", the front-end models that can be deployed independently (continuous models at the front of the queue) are selected. That is, starting from the head of the queue, it is judged in turn whether the model can be matched with an idle graphics card with sufficient video memory, until a certain model can no longer meet the condition of independent deployment on a single card. The previous models are the front-end models to be deployed, and the remaining models are the tail-end models to be deployed.

[0123] The memory allocation strategy for the front-end models to be deployed is a single-card dedicated memory strategy: each model is bound to an independent graphics card (prioritizing the allocation of the graphics card with the best compatibility and the strongest computing power), the upper limit of memory usage is set to its estimated peak memory usage, and resource isolation mechanisms (such as Docker memory limit, CUDA_VISIBLE_DEVICES lock) are used to avoid resource contention between models.

[0124] The memory allocation strategy for the tail-end models to be deployed is a time-sharing memory reuse strategy: memory usage windows are dynamically allocated based on model inference time and call frequency (e.g., models with high-frequency calls are allocated longer usage periods). Resource sharing is achieved through a model sleep-to-wake mechanism (memory is released when idle and loaded quickly when called). At the same time, a memory usage mutex lock is set to ensure no concurrent conflicts. If the server is a multi-GPU type, the tail-end models can also adopt a combination of cross-GPU shared memory and time-sharing reuse strategy to further improve resource utilization.

[0125] If the total cache deficit ΔTotal < the second cache threshold, meaning the server's total resources are insufficient:

[0126] Prioritize the deployment of low-priority models at the tail end of the model priority queue and perform memory compression optimization. Compression methods include: automatically reducing model inference accuracy, reducing batch size, enabling quantization optimization, and enabling gradient checkpointing. During the compression process, update the estimated peak memory usage of the model in real time and recalculate ΔTotal synchronously.

[0127] If ΔTotal is greater than or equal to the second cache threshold after one round of compression, video memory is allocated according to the above "front-end exclusive and tail-end reuse" strategy; if it is still not satisfied, compression is performed on low-priority models one by one in the order of "tail-end → middle-end" until ΔTotal reaches the target or the maximum number of compressions is reached (e.g., 3 times).

[0128] If ΔTotal is still less than the second cache threshold after reaching the maximum number of compressions, a resource shortage warning is triggered: a pop-up window on the front end prompts the user, displaying a "list of models that can release resources" (low-priority models) and "reasons why compression still cannot meet the requirements". Users can choose to "remove some low-priority models", "manually adjust the configuration of high-priority models (such as reducing precision)" or "terminate deployment". This step will be re-executed after the user confirms.

[0129] In scenarios with multiple GPUs of the same type, the time-sharing multiplexing strategy of the tail model can be combined with the efficient communication characteristics of the same card to centrally allocate the same type of model to some GPUs, reducing cross-card scheduling overhead.

[0130] In multi-GPU heterogeneous scenarios, the front-end high-priority model should be bound to the GPU with the best compatibility (such as deploying a PyTorch model on an NVIDIA GPU), while the time-sharing reuse of the tail model should avoid heterogeneous GPUs to avoid compatibility conflicts.

[0131] In single-GPU scenarios, only high-priority models are allowed to be exclusively used and low-priority models are allowed to be shared in a time-sharing manner. It is also necessary to ensure that the estimated peak value of each model’s video memory usage is less than or equal to 1.1 times the remaining video memory of the single card. If the condition is still not met after compression, the low-priority model will be forcibly removed.

[0132] This application ensures the exclusive use of resources and the performance of high-priority models, while achieving effective deployment of low-priority models through time-sharing multiplexing and video memory compression, and also takes into account the compatibility of different server types. It not only improves the utilization of video memory resources, but also ensures the reliability of core business models through priority adaptation and dynamic optimization, thereby improving the flexibility, stability and overall operating efficiency of multi-model deployment.

[0133] S307. Obtain preset compatibility rule information.

[0134] During the acquisition process, the system can first read the latest version of the local configuration repository and its preset compatibility rules. If a file is detected to be missing or corrupted, the system will automatically call the backup configuration file (stored in an off-site backup directory). If the backup file is also unavailable, the system will trigger a configuration anomaly warning and prompt the user to upload a standard configuration file, ensuring that subsequent dependency installation and environment configuration processes are supported by clear rules.

[0135] This step provides a core basis for multi-model dependency conflict detection and automated environment configuration through standardized and refined compatibility rules, avoiding deployment failures caused by dependency incompatibility and ensuring the automation and stability of the deployment process.

[0136] S308. If there is only one model to be deployed, then based on the preset compatibility rule information, install the dependency packages of each model to be deployed, obtain the model configuration file corresponding to the model to be deployed from the dependency package, and configure the model configuration file according to the video memory allocation strategy.

[0137] Prioritize matching and automatically installing the appropriate combination of dependency packages (such as CUDA, frameworks, and dedicated toolkits) from the local offline repository, and verify package integrity during the installation process.

[0138] Then, the default configuration file corresponding to the model (including weight path, inference parameters, etc.) is extracted. Based on the memory allocation strategy, key parameters such as target graphics card binding (e.g., setting CUDA_VISIBLE_DEVICES), memory usage limit, and service port are automatically configured. After the configuration is completed, model loading and service deployment are started.

[0139] S309. If there are multiple models to be deployed, the conflict status of the dependency package versions corresponding to each model to be deployed is checked and processed to obtain the conflict check result. Based on the memory allocation strategy, preset compatibility rule information and conflict check result, the multiple models to be deployed are deployed.

[0140] In some embodiments, if the conflict check result is no conflict, a unified environment for multiple models to be deployed is created, and in the unified environment, the models to be deployed are deployed sequentially according to the priority of each model and the memory allocation strategy and preset compatibility rule information.

[0141] Create a unified deployment environment, install compatible dependency packages in batches, deploy them sequentially according to model priority, and bind the graphics card resources specified by the video memory allocation strategy to avoid resource contention.

[0142] In some embodiments, if the conflict check result is that there is a conflict, an isolation environment corresponding to each model to be deployed is created; according to the priority of each model to be deployed, each model to be deployed is deployed in turn in the isolation environment corresponding to each model to be deployed, according to the memory allocation strategy and preset compatibility rule information.

[0143] Create independent isolated environments (Docker containers or virtual environments) for each model, install the corresponding dependency packages in each environment independently, and deploy them in the isolated environments in order of priority to avoid conflicts through environment isolation.

[0144] After deploying at least one model to be deployed, check whether the model deployment was successful, and perform a rollback operation if a failure is detected.

[0145] After the model is deployed, basic checks are automatically performed (such as calling test interfaces to verify service availability and checking whether the video memory / CPU usage is normal). If a deployment failure is detected (such as dependency installation errors or model loading failure), a rollback operation is immediately triggered: stop the relevant processes, uninstall the installed dependencies, release video memory / port resources, delete temporary configuration files, and record the reason for the failure for troubleshooting.

[0146] In this application, by adapting the deployment process according to the number of models and combining offline dependency installation and environment isolation mechanisms, deployment efficiency is ensured while avoiding dependency conflicts and resource contention issues; coupled with post-deployment verification and rollback, the stability and reliability of model deployment are further improved.

[0147] The model deployment method provided in this application can generate an accurate compatibility list by matching dedicated recommended models according to server type, support users to flexibly select models to be deployed, dynamically formulate the optimal video memory allocation strategy by combining model priority and hardware resources, achieve automatic offline dependency installation and conflict isolation with preset compatibility rules, and improve the adaptability, automation and resource utilization of model deployment by deployment verification and automatic rollback mechanism. At the same time, it avoids problems such as dependency conflicts and video memory overflow, reduces the cost of manual intervention and the risk of deployment failure, and ultimately improves the reliability, stability and overall efficiency of model deployment.

[0148] Figure 4This is a schematic diagram of the architecture of a model deployment device provided in an embodiment of this application. Please refer to [link / reference]. Figure 4 The model deployment device 400 may include a detection module 401, an analysis module 402, a determination module 403, and an execution module 404.

[0149] The detection module 401 is used to perform hardware environment detection on the server and obtain hardware detection information, including graphics card information and remaining resource information. The graphics card information includes the server type corresponding to the multiple graphics cards of the server and the remaining video memory of each graphics card.

[0150] The analysis module is used to analyze and process graphics card information and remaining resource information through the recommendation model corresponding to the server type, and obtain a model information list corresponding to at least one model to be deployed. The model information list includes the model identifier, model information and deployment configuration information corresponding to each model to be deployed.

[0151] The determination module 403 is used to determine the video memory allocation strategy corresponding to the model to be deployed based on the hardware detection information and the model information list corresponding to at least one model to be deployed.

[0152] The execution module 404 is used to execute the deployment of at least one model to be deployed based on the video memory allocation strategy.

[0153] In some embodiments, the analysis module is specifically used for:

[0154] Based on the recommended model corresponding to the server type, the hardware detection information is analyzed and processed to obtain a compatible recommendation list corresponding to the model to be deployed. The compatible recommendation list includes multiple recommended model identifiers and model information corresponding to each recommended model identifier.

[0155] Receive a list of model information corresponding to at least one model to be deployed selected by the user from the compatibility recommendation list.

[0156] In some embodiments, the analysis module is specifically used for:

[0157] If the server type is multi-GPU heterogeneous, then the recommendation model is a multi-GPU recommendation model. The hardware detection information is analyzed and processed through the multi-GPU recommendation model to obtain a compatible recommendation list.

[0158] If the server type is a single graphics card type or multiple graphics cards of the same type, the recommendation model is the single graphics card recommendation model. The hardware detection information is analyzed and processed through the single graphics card recommendation model to obtain a compatible recommendation list.

[0159] In some embodiments, the determining module 403 is specifically used for:

[0160] Based on the model information list corresponding to the model to be deployed, determine the estimated peak memory usage of each model to be deployed in at least one module to be deployed.

[0161] If there is only one model to be deployed, the memory allocation strategy is determined based on the estimated peak memory usage of the model to be deployed and the remaining memory of each graphics card.

[0162] If there are multiple models to be deployed, the memory allocation strategy is determined based on the priority of each model to be deployed, the estimated peak memory usage, and the remaining memory of each graphics card.

[0163] In some embodiments, the determining module 403 is specifically used for:

[0164] The difference between the remaining video memory of each card and the estimated peak video memory usage is calculated to obtain the cache space difference for each graphics card.

[0165] Determine if there are graphics cards whose cache space difference is greater than or equal to the first cache threshold;

[0166] If so, the video memory allocation strategy is a single-card independent video memory strategy;

[0167] If not, the memory allocation strategy is a multi-card collaborative deployment strategy.

[0168] In some embodiments, the determining module 403 is specifically used for:

[0169] Determine the model priority queues corresponding to multiple models to be deployed. The model priority queues are arranged from high to low priority of each model to be deployed.

[0170] The sum of the remaining video memory of each graphics card is used to determine the total remaining video memory of the multiple graphics cards.

[0171] Based on the estimated peak memory usage of each model to be deployed, determine the total memory usage corresponding to multiple models to be deployed.

[0172] Determine the total difference between the total remaining video memory and the total video memory usage in the cache.

[0173] Determine whether the total cache difference is greater than or equal to the second cache threshold;

[0174] If so, the model priority queue is divided into front-end models to be deployed and tail-end models to be deployed; the memory allocation strategy for the front-end models to be deployed is determined to be a single-card independent memory strategy; the memory allocation strategy for the tail-end models to be deployed is determined to be a memory time-sharing strategy.

[0175] If not, then the models to be deployed at the end of the model priority queue will be subjected to memory compression to ensure that the total cache difference is greater than or equal to the second cache threshold.

[0176] In some embodiments, execution module 404 is specifically used for:

[0177] Obtain preset compatibility rule information;

[0178] If there is only one model to be deployed, the dependency packages of each model to be deployed will be installed based on the preset compatibility rule information, and the model configuration file corresponding to the model to be deployed will be obtained from the dependency package. The model configuration file will be configured according to the video memory allocation strategy.

[0179] If there are multiple models to be deployed, the conflict status of the dependency package versions corresponding to each model to be deployed is checked and the conflict check result is obtained; based on the memory allocation strategy, preset compatibility rule information and conflict check result, the multiple models to be deployed are deployed.

[0180] In some embodiments, execution module 404 is specifically used for:

[0181] If the conflict check result is no conflict, then create a unified environment for multiple models to be deployed, and in the unified environment, deploy the models to be deployed in sequence according to the priority of each model and the memory allocation strategy and preset compatibility rules.

[0182] If the conflict check result indicates a conflict, an isolation environment is created for each model to be deployed. Based on the priority of each model to be deployed, each model is deployed sequentially in the isolation environment corresponding to each model, according to the memory allocation strategy and preset compatibility rules.

[0183] In some embodiments, the apparatus further includes a rollback module:

[0184] The rollback module is used to detect whether the model deployment was successful and to perform a rollback operation when a failure is detected.

[0185] The model deployment device provided in this application embodiment can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0186] Figure 5 This is a schematic diagram of the structure of an electronic device provided in this application. Figure 5 As shown, the electronic device 500 provided in this embodiment includes at least one processor 501 and a memory 502. Optionally, the electronic device 500 further includes a communication component 503. The processor 501, memory 502, and communication component 503 are connected via a bus.

[0187] In the specific implementation process, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to execute the above-described model deployment method embodiment.

[0188] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0189] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0190] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0191] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0192] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described model deployment method embodiments at runtime.

[0193] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0194] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described model deployment method embodiments.

[0195] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described model deployment method embodiments.

[0196] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0197] The above provides a detailed description of a model deployment method provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A model deployment method, characterized in that, include: The server undergoes hardware environment testing to obtain hardware testing information, which includes graphics card information and remaining resource information. The graphics card information includes the server type corresponding to the multiple graphics cards of the server, as well as the remaining video memory of each graphics card. By using the recommendation model corresponding to the server type, the graphics card information and the remaining resource information are analyzed and processed to obtain a model information list corresponding to at least one model to be deployed. The model information list includes the model identifier, model information and deployment configuration information corresponding to each model to be deployed. Based on the hardware detection information and the model information list corresponding to the at least one model to be deployed, the video memory allocation strategy corresponding to the model to be deployed is determined; Based on the aforementioned memory allocation strategy, the deployment of at least one model to be deployed is executed.

2. The method according to claim 1, characterized in that, The hardware detection information is analyzed and processed using the recommendation model corresponding to the server type to obtain a list of model information corresponding to at least one model to be deployed, including: Based on the recommended model corresponding to the server type, the hardware detection information is analyzed and processed to obtain a compatible recommendation list corresponding to the model to be deployed. The compatible recommendation list includes multiple recommended model identifiers and model information corresponding to each recommended model identifier. Receive a list of model information corresponding to at least one model to be deployed selected by the user from the compatibility recommendation list.

3. The method according to claim 2, characterized in that, Based on the recommendation model corresponding to the server type, the hardware detection information is analyzed and processed to obtain a compatibility recommendation list corresponding to the model to be deployed, including: If the server type is a multi-graphics card heterogeneous type, then the recommendation model is a multi-graphics card recommendation model. The hardware detection information is analyzed and processed through the multi-graphics card recommendation model to obtain the compatibility recommendation list. If the server type is a single graphics card type or multiple graphics cards of the same type, then the recommendation model is a single-type graphics card recommendation model. The hardware detection information is analyzed and processed through the single-type graphics card recommendation model to obtain the compatibility recommendation list.

4. The method according to claim 1, characterized in that, Based on the hardware detection information and the model information list corresponding to the at least one model to be deployed, determine the video memory allocation strategy corresponding to the model to be deployed, including: Based on the model information list corresponding to the model to be deployed, determine the estimated peak memory usage of each model to be deployed in the at least one module to be deployed; If there is only one model to be deployed, the video memory allocation strategy is determined based on the estimated peak video memory usage of the model to be deployed and the remaining video memory of each graphics card. If there are multiple models to be deployed, the video memory allocation strategy is determined based on the priority of each model to be deployed, the estimated peak video memory usage, and the remaining video memory of each graphics card.

5. The method according to claim 4, characterized in that, Based on the estimated peak video memory usage of the model to be deployed and the remaining video memory of each graphics card, the video memory allocation strategy is determined, including: The difference between the remaining video memory of each card and the estimated peak video memory usage is calculated to obtain the cache space difference for each graphics card. Determine whether there is a graphics card whose cache space difference is greater than or equal to the first cache threshold; If so, then the video memory allocation strategy is a single-card independent video memory strategy; If not, then the memory allocation strategy is a multi-card collaborative deployment strategy.

6. The method according to claim 4, characterized in that, Based on the priority of each model to be deployed, the estimated peak memory usage, and the remaining memory on each graphics card, the memory allocation strategy is determined, including: A model priority queue is determined for multiple models to be deployed, wherein the model priority queue is arranged from high to low priority of each model to be deployed; The sum of the remaining video memory of each graphics card is determined as the total remaining video memory of the multiple graphics cards. Based on the estimated peak memory usage of each model to be deployed, the total memory usage corresponding to the multiple models to be deployed is determined. Determine the total difference between the total remaining video memory and the total video memory usage in the cache. Determine whether the total cache difference is greater than or equal to the second cache threshold; If so, the model priority queue is divided into a front-end model to be deployed and a tail-end model to be deployed; the memory allocation strategy for the front-end model to be deployed is determined to be a single-card independent memory strategy; and the memory allocation strategy for the tail-end model to be deployed is determined to be a memory time-sharing strategy. If not, then the models to be deployed at the tail end of the model priority queue will be subjected to memory compression to ensure that the total cache difference is greater than or equal to the second cache threshold.

7. The method according to claim 1, characterized in that, Based on the aforementioned memory allocation strategy, the deployment of the at least one model to be deployed is performed, including: Obtain preset compatibility rule information; If there is only one model to be deployed, then based on the preset compatibility rule information, the dependency packages of each model to be deployed are installed, and the model configuration file corresponding to the model to be deployed is obtained from the dependency package. The model configuration file is configured according to the video memory allocation strategy. If there are multiple models to be deployed, the conflict status of the dependency package versions corresponding to each model to be deployed is verified to obtain the conflict verification result; according to the memory allocation strategy, the preset compatibility rule information and the conflict verification result, the multiple models to be deployed are deployed.

8. The method according to claim 7, characterized in that, Based on the aforementioned memory allocation strategy, preset compatibility rule information, and conflict verification results, multiple models to be deployed are deployed, including: If the conflict check result is no conflict, then multiple unified environments for the models to be deployed are created, and in the unified environment, the models to be deployed are deployed sequentially according to the priority of each model and the memory allocation strategy and the preset compatibility rule information. If the conflict check result indicates a conflict, an isolation environment is created for each model to be deployed. Based on the priority of each model to be deployed, the models are deployed sequentially in the isolation environment corresponding to each model to be deployed, according to the memory allocation strategy and preset compatibility rules.

9. The method according to claim 1, characterized in that, After deploying the at least one model to be deployed, the process further includes: The system checks whether the model deployment was successful and performs a rollback operation if a failure is detected.

10. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-9.

Images