Efficient deployment method and system for privatized large language model ecological service

Abstract

The invention relates to an efficient deployment method and system for a privatized large language model ecological service. The system comprises an evaluation module, an index collector, a monitoring module, a vLLM framework, a model cluster service module and a request distribution module. The evaluation module performs quantitative evaluation on the performance of the system under different hardware platforms; the index collector collects server cluster performance indexes, cluster resource states and load information in real time; the monitoring module is used for monitoring the server cluster; the vLLM framework dynamically allocates computing resources to realize management of a multi-modal server cluster, and performs unified management and scheduling through PM2 to realize multi-model efficient parallel task processing; a vLLM framework is started by the model cluster service module, and different large model instances are packaged in a Docker container; the request distribution module distributes external requests in a self-adaptive mode according to cluster resource states and load information fed back by the index collector in real time. Intelligent request distribution and dynamic load balancing are realized, and integration and efficient management of a multi-modal large language model are realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of large language model deployment, and in particular, relates to an efficient deployment method and system for privatized large language model ecological services. Background Art

[0002] Under the current technical background, the deployment of private large language models faces many challenges. First, the construction efficiency of existing multimodal large language model clusters is low, and there is a lack of automated and intelligent scheduling strategies, which leads to low resource utilization and makes it difficult to fully utilize the computing potential of hardware devices. Secondly, when processing complex external requests, the load balancing mechanism of existing technologies is relatively simple and cannot dynamically adapt to request fluctuations and cluster resource distribution, resulting in system response delays and performance bottlenecks. In addition, the existing large language model cluster monitoring methods are mostly passive, making it difficult to capture cluster performance changes and potential failure risks in a timely manner, and unable to provide efficient disaster recovery and high availability support. Most importantly, the current model deployment solutions often fail to fully consider the differences in various hardware environments, and lack deployment optimization solutions based on resource consumption and hardware capabilities, making it impossible to uniformly optimize the model performance in single graphics card, multi-graphics card or multi-server environments, resulting in high deployment costs and low efficiency.

[0003] The patent is called a high-concurrency large language model high-speed reasoning deployment method. It uses Docker containers to deploy large language models on a GPU cluster composed of multiple physical machines to ensure the environmental consistency and efficient use of each machine. It uses the vLLM open source framework and BFloat16 floating point format to improve the reasoning deployment efficiency, support large-scale concurrent processing, screen out the best post-processing parameters through testing, build a streaming reasoning API, and achieve real-time reasoning of large language models in a high-concurrency environment. However, the patent cannot achieve request distribution and dynamic load balancing, integration and efficient management of multi-modal large language models, real-time performance monitoring and disaster recovery mechanisms, and model performance evaluation and dynamic optimization.

[0004] Therefore, how to improve the existing large language model deployment method to achieve intelligent request distribution and dynamic load balancing, integration and efficient management of multimodal large language models, real-time performance monitoring and disaster recovery mechanism, optimized deployment of various hardware environments, and model performance evaluation and dynamic optimization are technical problems that need to be solved urgently. Summary of the invention

[0005] The purpose of the present invention is to provide an efficient deployment method and system for a privatized large language model ecological service, so as to realize intelligent request distribution and dynamic load balancing, integration and efficient management of multimodal large language models, real-time performance monitoring and disaster recovery mechanism, optimized deployment of various hardware environments, and model performance evaluation and dynamic optimization.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0007] An efficient deployment system for privatized large language model ecological services, comprising an evaluation module, an indicator collector, a monitoring module, a vLLM framework, a model cluster service module and a request distribution module; the evaluation module is connected to the model cluster service module, the indicator collector and the monitoring module are both connected to the model cluster service module through the vLLM framework, the indicator collector and the monitoring module are both connected to the request distribution module through a visualization panel, and the request distribution module is connected to the model cluster service module through a custom request port;

[0008] The evaluation module quantitatively evaluates the performance of the system on different hardware platforms by setting lm-eval and vLLM Benchmark tools;

[0009] The indicator collector is used to collect the specified performance indicators of the server cluster and the cluster resource status and load information in real time;

[0010] The monitoring module is used to monitor the server cluster based on the real-time collection of various specified performance indicators of the server cluster and the cluster resource status and load information by the indicator collector;

[0011] The vLLM framework is used to manage multimodal server clusters through intelligent dynamic allocation of computing resources, and to achieve efficient parallel processing of multi-model tasks through unified management and scheduling through PM2;

[0012] The model cluster service module starts the vLLM framework through PM2, encapsulates different large model instances in Docker containers, collects the spatial information of all current graphics cards, and selects the optimal configuration solution based on the spatial information of the graphics cards;

[0013] The request distribution module is used to adaptively distribute external requests according to the cluster resource status and load information fed back in real time by the indicator collector, and automatically route to suitable model instances based on different request types.

[0014] Preferably, the vLLM framework includes a large language model, a large language model engine, a cache engine, and a model training module, wherein the large language model is connected to the large language model engine, the large language model engine is connected to the cache engine, and the large language model is connected to the model training module;

[0015] The large language model predicts the load data of the server cluster and controls the entire reasoning process and pre-allocation of load data for top-level user applications;

[0016] The large language model engine includes a task scheduling management module and a thread module. The task scheduling management module is used to schedule different requests to implement the cache block resources within the vLLM framework to complete the execution of existing requests;

[0017] The cache engine implements the large language model reasoning of the server cluster CPU or GPU, and executes the scheduling of the corresponding cache block resources requested by the task scheduling management module;

[0018] The model training module is used to train and verify the large language model.

[0019] Preferably, the large language model engine schedules different requests to implement the cache block resources within the vLLM framework to complete the execution of existing requests as follows:

[0020] First, the task scheduling is initialized. The load value of each server is calculated based on the hardware resource data of the server cluster and compared with the preset load threshold. If the load of the server is greater than the load threshold, its weight is reduced. If the load of the server is less than the load threshold, its weight is increased.

[0021] Then, during the task scheduling process, polling scheduling is performed according to the weights of the initialized servers, and more requests are scheduled for servers with larger weights;

[0022] After each scheduling, the corresponding weight is updated according to the actual load of the server, and as the server's ability to process requests changes due to task scheduling, its weight is updated and adjusted accordingly after the request processing capacity changes.

[0023] Preferably, the large language model comprises an input layer, a multi-layer encoder, a multi-layer decoder and an output layer, wherein the input layer is connected to the encoder, the encoder is connected to the decoder, and the decoder is connected to the output layer;

[0024] The input layer is used to input the specified performance indicators of the server cluster and the cluster resource status and load information collected in real time by the indicator collector, and convert the input text data into a continuous representation vector of a specified dimension, wherein the continuous representation vector is the vector sum of the text data word vector and the text data position vector;

[0025] Each encoder layer in the multi-layer encoder includes a self-attention layer and a feed-forward neural network layer, wherein the self-attention layer captures the dependency relationship in the text based on calculating the correlation score between different positions in the input text sequence, and the feed-forward neural network performs a nonlinear transformation on the output of the self-attention layer, and the feed-forward neural network includes two linear layers and an activation function;

[0026] Each decoder layer of the multi-layer decoder includes a self-attention layer, a cross-attention layer and a feedforward neural network layer. The cross-attention layer is used to extract the output information of the encoder and the information of the position of the current layer decoder, so that the large language model can understand the content of the entire input text sequence.

[0027] Preferably, the working process of the encoder is:

[0028] The self-attention layer of the encoder performs self-attention calculation on the correlation between each text word input by the input layer and all other text words, and obtains the correlation score between each text word and all other text words;

[0029] Decompose the input text sequence into multiple heads, and calculate the relevance score between each text word and all other text words on each head;

[0030] The representation vector of each text word is processed nonlinearly by two linear layers and an activation function of a feedforward neural network to obtain the corresponding encoding information vector generated by each word in the input text sequence.

[0031] Preferably, the encoder generates a corresponding encoding information vector for each word in the obtained input text sequence and inputs it into the decoder. The decoder generates a target sequence based on the encoding information vector and processes the representation vector of the text word through a feedforward neural network. The decoder generates a representation vector sequence, and each vector in the representation vector sequence corresponds to a position in the target sequence. Each vector is processed by a linear layer and a softmax function to generate a probability distribution of the vocabulary size.

[0032] In a second aspect, a method for efficiently deploying a privatized large language model ecological service is provided, and the method for efficiently deploying a privatized large language model ecological service based on any one of the above methods includes the following processes:

[0033] S1: Deploy the multimodal evaluation module, indicator collector, monitoring module, vLLM framework, model cluster service module, and request distribution module to connect the evaluation module with the model cluster service module. The indicator collector and monitoring module are connected to the model cluster service module through the vLLM framework. The indicator collector and monitoring module are connected to the request distribution module through the visualization panel. The request distribution module is connected to the model cluster service module through the custom request port.

[0034] S2: Start the vLLM framework through PM2 to encapsulate and manage different multimodal large model instances to achieve independent, multi-card and distributed deployment. At the same time, PM2 performs process guarding, disaster recovery management, automatic restart and performance monitoring.

[0035] S3: The large language module in the vLLM framework predicts the server load data, and the request distribution module performs pre-allocation based on the predicted server load data;

[0036] S4: The indicator collector collects various specified performance indicators of the server cluster and the cluster resource status and load information in real time, and the monitoring module monitors the server cluster based on the various specified performance indicators of the server cluster and the cluster resource status and load information collected by the indicator collector in real time;

[0037] S5: The request distribution module adjusts the pre-allocated load according to the cluster resource status and load information fed back in real time by the indicator collector and the monitoring data of the monitoring module, and adaptively distributes external requests. At the same time, it automatically routes them to the appropriate model instance based on different request types.

[0038] The beneficial effects of the present invention include:

[0039] The invention provides an efficient deployment method and system for a private large language model ecological service, including an evaluation module, an indicator collector, a monitoring module, a vLLM framework, a model cluster service module, and a request distribution module. The evaluation module quantitatively evaluates the performance of the system under different hardware platforms; the indicator collector collects server cluster performance indicators, cluster resource status, and load information in real time; the monitoring module monitors the server cluster; the vLLM framework dynamically allocates computing resources to achieve management of multimodal server clusters, and implements efficient parallel processing of multi-model tasks through unified management and scheduling through PM2; the model cluster service module starts the vLLM framework and encapsulates different large model instances in Docker containers; the request distribution module adaptively distributes external requests based on the cluster resource status and load information fed back in real time by the indicator collector. Intelligent request distribution and dynamic load balancing, integration and efficient management of multimodal large language models are realized.

[0040] First, the vLLM framework sets up a large language model to predict the load data of the server cluster, and for the top-level user application, it controls the entire reasoning process and pre-allocation of load data. Then, combined with the subsequent indicator data and monitoring data based on actual collection, the pre-allocation of load data is adjusted to ultimately achieve accurate scheduling of each task.

[0041] Secondly, through the initialization of task scheduling, the load value of each server is calculated according to the hardware resource data of the server cluster, and compared with the preset load threshold. If the load of the server is greater than the load threshold, its weight is reduced; if the load of the server is less than the load threshold, its weight is increased; in the process of task scheduling, polling scheduling is performed according to the weight of the initialized server, and more requests are scheduled for servers with larger weights; after each scheduling, the corresponding weight is updated according to the actual load of the server, and as the server's ability to process requests changes due to task scheduling, its weight is updated and adjusted accordingly after the request processing capacity changes. This further improves the intelligence and precision of task scheduling.

[0042] Thirdly, the large language model includes an input layer, a multi-layer encoder, a multi-layer decoder, and an output layer. The input layer input indicator collector collects the specified performance indicators of the server cluster in real time, as well as the cluster resource status and load information. Each encoder package in the multi-layer encoder captures the dependencies in the text through the self-attention layer; each decoder in the multi-layer decoder extracts the output information of the encoder and the information of the position of the current layer decoder, so that the large language model understands the content of the entire input text sequence. Accurate prediction of the large language model is achieved.

[0043] Finally, multiple large language models and multimodal processing models are integrated through the vLLM framework, and PM2 is used to implement parallel management and resource scheduling of model instances, effectively improving the operating efficiency of the model cluster. The indicator collector and monitoring module are integrated to achieve real-time performance monitoring of all model instances in the cluster, support disaster recovery mechanism, and ensure stable operation of the system under high load or abnormal conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A schematic diagram of the composition structure of the efficient deployment system for the privatized large language model ecosystem service of the present invention.

[0045] Figure 2 It is a schematic diagram of the composition structure of the privatized large language model of the present invention.

[0046] Figure 3 A flowchart of an efficient deployment method for a private large language model ecosystem service of the present invention. DETAILED DESCRIPTION

[0047] The following is combined with Figure 1 to Figure 3 The present invention is further described in detail:

[0048] Example 1

[0049] See attached Figure 1As shown, an efficient deployment system for privatized large language model ecological services includes an evaluation module, an indicator collector, a monitoring module, a vLLM framework, a model cluster service module and a request distribution module; the evaluation module is connected to the model cluster service module, the indicator collector and the monitoring module are connected to the model cluster service module through the vLLM framework, the indicator collector and the monitoring module are connected to the request distribution module through a visualization panel, and the request distribution module is connected to the model cluster service module through a custom request port. The vLLM framework is the core of the entire architecture and is mainly responsible for managing the efficient operation of multimodal large model clusters. The framework solves the common resource waste problem in the large model deployment process by dynamically allocating computing resources, especially GPU memory. vLLM allows multiple large model instances to share the same GPU resources, thereby maximizing the utilization of hardware resources. The vLLM framework is uniformly managed and scheduled through PM2 to ensure the efficiency of multi-model parallel processing tasks.

[0050] The evaluation module uses lm-eval and vLLM Benchmark tools to quantitatively evaluate the performance of the system on different hardware platforms. It mainly uses quantitative methods for different hardware platforms to help evaluate the performance of the model under different hardware configurations. Through these evaluation tools, the system can automatically generate the optimal configuration solution for a single graphics card, multiple graphics cards, or multiple server environments, and automatically or manually decide whether to retain certain model instances based on the actual performance of the model.

[0051] The indicator collector collects the specified performance indicators of the server cluster and the cluster resource status and load information in real time. The monitoring module monitors the server cluster based on the specified performance indicators of the server cluster and the cluster resource status and load information collected in real time by the indicator collector; the vLLM framework realizes the management of the multimodal server cluster by intelligently and dynamically allocating computing resources, and realizes efficient parallel processing of multiple models through unified management and scheduling through PM2; the model cluster service module starts the vLLM framework through PM2, encapsulates different large model instances in Docker containers, collects the space information of all current graphics cards, and selects the optimal configuration plan based on the space information of the graphics cards; the request distribution module is used to adaptively distribute external requests according to the cluster resource status and load information fed back by the indicator collector in real time, and automatically routes to the appropriate model instance based on different request types.

[0052] Example 2

[0053] On the basis of Example 1, the vLLM framework includes a large language model, a large language model engine, a cache engine, and a model training module. The large language model is connected to the large language model engine, the large language model engine is connected to the cache engine, and the large language model is connected to the model training module. The large language model predicts the load data of the server cluster and controls the entire reasoning process and the pre-allocation of load data for top-level user applications. The large language model engine includes a task scheduling management module and a thread module. The task scheduling management module is used to schedule different requests and implement the cache block resources within the vLLM framework to complete the execution of existing requests. The cache engine implements the large language model reasoning of the CPU or GPU of the server cluster and executes the task scheduling management module to schedule the corresponding cache block resources of the request; the model training module is used to train and verify the large language model.

[0054] In this embodiment, the large language model engine schedules different requests to implement the cache block resources within the vLLM framework to complete the execution of existing requests as follows:

[0055] First, the task scheduling is initialized. The load value of each server is calculated based on the hardware resource data of the server cluster and compared with the preset load threshold. If the load of the server is greater than the load threshold, its weight is reduced. If the load of the server is less than the load threshold, its weight is increased.

[0056] Then, during the task scheduling process, polling scheduling is performed according to the weights of the initialized servers, and more requests are scheduled for servers with larger weights;

[0057] After each scheduling, the corresponding weight is updated according to the actual load of the server, and as the server's ability to process requests changes due to task scheduling, its weight is updated and adjusted accordingly after the request processing capacity changes.

[0058] This process is the underlying logic of task scheduling. The initialization of task scheduling calculates the load value of each server according to the hardware resource data of the server cluster for initial task scheduling, and then compares it with the preset load threshold to adjust the weight. Polling scheduling is performed according to the weight of the initialized server, and more requests are scheduled for servers with larger weights. After each scheduling, the corresponding weight is updated according to the actual load of the server. As the server's ability to process requests changes due to task scheduling, its weight is updated and adjusted accordingly after the request processing capacity changes.

[0059] Example 3

[0060] In Example 1 or Example 2, see Figure 2The large language model includes an input layer, a multi-layer encoder, a multi-layer decoder and an output layer, wherein the input layer is connected to the encoder, the encoder is connected to the decoder, and the decoder is connected to the output layer. The input layer is used to input the specified performance indicators of the server cluster collected in real time by the indicator collector, as well as the cluster resource status and load information, and convert the input text data into a continuous representation vector of a specified dimension, wherein the continuous representation vector is the vector sum of the text data word vector and the text data position vector.

[0061] Each encoder layer in the multi-layer encoder includes a self-attention layer and a feedforward neural network layer, wherein the self-attention layer captures the dependency relationship in the text based on calculating the correlation score between different positions in the input text sequence, and the feedforward neural network performs a nonlinear transformation on the output of the self-attention layer, and the feedforward neural network includes two linear layers and an activation function. Each decoder layer of the multi-layer decoder includes a self-attention layer, a cross-attention layer and a feedforward neural network layer, and the cross-attention layer is used to extract the output information of the encoder and the information of the decoder position of the current layer, so that the large language model understands the content of the entire input text sequence.

[0062] In this embodiment, the working process of the encoder is:

[0063] The self-attention layer of the encoder performs self-attention calculation on the correlation between each text word input through the input layer and all other text words, and obtains the correlation score between each text word and all other text words. The input text sequence is decomposed into multiple heads, and the correlation score between each text word and all other text words is calculated on each head. The representation vector of each text word is processed nonlinearly through two linear layers and activation functions of the feedforward neural network to obtain the corresponding encoding information vector generated by each word in the input text sequence.

[0064] The encoder generates a corresponding encoding information vector for each word in the obtained input text sequence and inputs it into the decoder. The decoder generates a target sequence based on the encoding information vector and processes the representation vector of the text word through a feedforward neural network. The decoder generates a representation vector sequence. Each vector in the representation vector sequence corresponds to a position in the target sequence. Each vector is processed through a linear layer and a softmax function to generate a probability distribution of the vocabulary size.

[0065] An efficient deployment method for a privatized large language model ecosystem service, based on any of the above-mentioned efficient deployment methods for a privatized large language model ecosystem service, see Figure 3 , including the following processes:

[0066] S1: Deploy the multimodal evaluation module, indicator collector, monitoring module, vLLM framework, model cluster service module, and request distribution module to connect the evaluation module with the model cluster service module. The indicator collector and monitoring module are connected to the model cluster service module through the vLLM framework. The indicator collector and monitoring module are connected to the request distribution module through the visualization panel. The request distribution module is connected to the model cluster service module through the custom request port.

[0067] S2: The vLLM framework is started through PM2 to encapsulate and manage different multimodal large model instances to achieve independent, multi-card and distributed deployment. At the same time, PM2 performs process guardianship, disaster recovery management, automatic restart and performance monitoring. By combining vLLM and PM2 technologies, the present invention can quickly build and combine multimodal large language model clusters, and through intelligent scheduling strategies, dynamically allocate computing resources to improve the overall computing efficiency and hardware utilization of the cluster.

[0068] S3: The large language module in the vLLM framework predicts the server load data, and the request distribution module performs pre-allocation based on the predicted server load data;

[0069] S4: The indicator collector collects various specified performance indicators of the server cluster and the cluster resource status and load information in real time, and the monitoring module monitors the server cluster based on the various specified performance indicators of the server cluster and the cluster resource status and load information collected by the indicator collector in real time;

[0070] S5: The request distribution module adjusts the pre-allocated load according to the cluster resource status and load information fed back in real time by the indicator collector and the monitoring data of the monitoring module, and adaptively distributes external requests. At the same time, it automatically routes to the appropriate model instance based on different request types. It can dynamically adjust the distribution strategy according to the resource status of the cluster and the request load to ensure rapid system response, avoid performance bottlenecks, and improve the overall service quality. By collecting public and private data sets, a comprehensive performance evaluation baseline is established, and the performance of the privatized large language model cluster can be quickly analyzed while ensuring data privacy and security.

[0071] In summary, the present invention provides a method and system for efficiently deploying a privatized large language model ecological service. The vLLM framework sets a large language model, predicts the load data of the server cluster, and controls the entire reasoning process and the pre-allocation of load data for top-level user applications. Then, combined with subsequent indicator data and monitoring data based on actual collection, the pre-allocation of load data is adjusted, and finally the precise scheduling of each task is achieved. Through the initialization of task scheduling, the load value of each server is calculated according to the hardware resource data of the server cluster, and compared with the preset load threshold. If the load of the server is greater than the load threshold, its weight is reduced, and if the load of the server is less than the load threshold, its weight is increased; in the process of task scheduling, polling scheduling is performed according to the weight of the initialized server, and more requests are scheduled for servers with larger weights; after each scheduling, the corresponding weight is updated according to the actual load of the server, and as the server's ability to process requests changes due to task scheduling, its weight is updated and adjusted accordingly after the processing request ability changes. Further improve the intelligence and precision of task scheduling.

[0072] The large language model includes an input layer, a multi-layer encoder, a multi-layer decoder, and an output layer. The input layer input indicator collector collects the specified performance indicators of the server cluster in real time, as well as the cluster resource status and load information. Each encoder packet in the multi-layer encoder captures the dependencies in the text through the self-attention layer; each decoder in the multi-layer decoder extracts the output information of the encoder and the information of the position of the decoder in the current layer, so that the large language model understands the content of the entire input text sequence. Accurate prediction of the large language model is achieved. Multiple large language models and multimodal processing models are integrated through the vLLM framework, and the parallel management and resource scheduling of model instances are realized through PM2, which effectively improves the operating efficiency of the model cluster. The indicator collector and monitoring module are integrated to realize real-time performance monitoring of all model instances in the cluster, support disaster recovery mechanism, and ensure the stable operation of the system under high load or abnormal conditions.

Claims

1. An efficient deployment system for privatized large language model ecosystem services, characterized in that: It includes an evaluation module, an indicator collector, a monitoring module, a vLLM framework, a model cluster service module and a request distribution module; the evaluation module is connected to the model cluster service module, the indicator collector and the monitoring module are both connected to the model cluster service module through the vLLM framework, the indicator collector and the monitoring module are both connected to the request distribution module through a visualization panel, and the request distribution module is connected to the model cluster service module through a custom request port; The evaluation module quantitatively evaluates the performance of the system on different hardware platforms by setting lm-eval and vLLM Benchmark tools; The indicator collector is used to collect the specified performance indicators of the server cluster and the cluster resource status and load information in real time; The monitoring module is used to monitor the server cluster based on the real-time collection of various specified performance indicators of the server cluster and the cluster resource status and load information by the indicator collector; The vLLM framework is used to manage multimodal server clusters through intelligent dynamic allocation of computing resources, and to achieve efficient parallel processing of multi-model tasks through unified management and scheduling through PM2; The model cluster service module starts the vLLM framework through PM2, encapsulates different large model instances in Docker containers, collects the spatial information of all current graphics cards, and selects the optimal configuration solution based on the spatial information of the graphics cards; The request distribution module is used to adaptively distribute external requests according to the cluster resource status and load information fed back in real time by the indicator collector, and automatically route to suitable model instances based on different request types.

2. According to claim 1, an efficient deployment system for a privatized large language model ecosystem service is characterized in that: The vLLM framework includes a large language model, a large language model engine, a cache engine, and a model training module, wherein the large language model is connected to the large language model engine, the large language model engine is connected to the cache engine, and the large language model is connected to the model training module; The large language model predicts the load data of the server cluster and controls the entire reasoning process and pre-allocation of load data for top-level user applications; The large language model engine includes a task scheduling management module and a thread module. The task scheduling management module is used to schedule different requests to implement the cache block resources within the vLLM framework to complete the execution of existing requests; The cache engine implements the large language model reasoning of the server cluster CPU or GPU, and executes the scheduling of the corresponding cache block resources requested by the task scheduling management module; The model training module is used to train and verify the large language model.

3. According to claim 2, an efficient deployment system for a private large language model ecosystem service is characterized in that: The process of the large language model engine scheduling different requests to implement the cache block resources in the vLLM framework to complete the execution of existing requests is as follows: First, the task scheduling is initialized. The load value of each server is calculated based on the hardware resource data of the server cluster and compared with the preset load threshold. If the load of the server is greater than the load threshold, its weight is reduced. If the load of the server is less than the load threshold, its weight is increased. Then, during the task scheduling process, polling scheduling is performed according to the weights of the initialized servers, and more requests are scheduled for servers with larger weights; After each scheduling, the corresponding weight is updated according to the actual load of the server, and as the server's ability to process requests changes due to task scheduling, its weight is updated and adjusted accordingly after the request processing capacity changes.

4. According to claim 2, an efficient deployment system for a private large language model ecosystem service is characterized in that: The large language model comprises an input layer, a multi-layer encoder, a multi-layer decoder and an output layer, wherein the input layer is connected to the encoder, the encoder is connected to the decoder, and the decoder is connected to the output layer; The input layer is used to input the specified performance indicators of the server cluster and the cluster resource status and load information collected in real time by the indicator collector, and convert the input text data into a continuous representation vector of a specified dimension, wherein the continuous representation vector is the vector sum of the text data word vector and the text data position vector; Each encoder layer in the multi-layer encoder includes a self-attention layer and a feed-forward neural network layer, wherein the self-attention layer captures the dependency relationship in the text based on calculating the correlation score between different positions in the input text sequence, and the feed-forward neural network performs a nonlinear transformation on the output of the self-attention layer, and the feed-forward neural network includes two linear layers and an activation function; Each decoder layer of the multi-layer decoder includes a self-attention layer, a cross-attention layer and a feedforward neural network layer. The cross-attention layer is used to extract the output information of the encoder and the information of the position of the current layer decoder, so that the large language model can understand the content of the entire input text sequence.

5. According to claim 4, an efficient deployment system for a private large language model ecological service is characterized in that: The working process of the encoder is: The self-attention layer of the encoder performs self-attention calculation on the correlation between each text word input by the input layer and all other text words, and obtains the correlation score between each text word and all other text words; Decompose the input text sequence into multiple heads, and calculate the relevance score between each text word and all other text words on each head; The representation vector of each text word is processed nonlinearly by two linear layers and an activation function of a feedforward neural network to obtain the corresponding encoding information vector generated by each word in the input text sequence.

6. According to claim 5, an efficient deployment system for a private large language model ecological service is characterized in that: The encoder generates a corresponding encoding information vector for each word in the obtained input text sequence and inputs it into the decoder. The decoder generates a target sequence based on the encoding information vector and processes the representation vector of the text word through a feedforward neural network. The decoder generates a representation vector sequence. Each vector in the representation vector sequence corresponds to a position in the target sequence. Each vector is processed through a linear layer and a softmax function to generate a probability distribution of the vocabulary size.

7. An efficient deployment method for a privatized large language model ecological service, based on an efficient deployment method for a privatized large language model ecological service according to any one of claims 1 to 6, characterized in that: The process includes: S1: Deploy the multimodal evaluation module, indicator collector, monitoring module, vLLM framework, model cluster service module, and request distribution module to connect the evaluation module with the model cluster service module. The indicator collector and monitoring module are connected to the model cluster service module through the vLLM framework. The indicator collector and monitoring module are connected to the request distribution module through the visualization panel. The request distribution module is connected to the model cluster service module through the custom request port. S2: Start the vLLM framework through PM2 to encapsulate and manage different multimodal large model instances to achieve independent, multi-card and distributed deployment. At the same time, PM2 performs process guarding, disaster recovery management, automatic restart and performance monitoring. S3: The large language module in the vLLM framework predicts the server load data, and the request distribution module performs pre-allocation based on the predicted server load data; S4: The indicator collector collects various specified performance indicators of the server cluster and the cluster resource status and load information in real time, and the monitoring module monitors the server cluster based on the various specified performance indicators of the server cluster and the cluster resource status and load information collected by the indicator collector in real time; S5: The request distribution module adjusts the pre-allocated load according to the cluster resource status and load information fed back in real time by the indicator collector and the monitoring data of the monitoring module, and adaptively distributes external requests. At the same time, it automatically routes them to the appropriate model instance based on different request types.

Images