Method and system for expanding and shrinking capacity based on large model

Abstract

The invention discloses a capacity expansion and contraction method and system based on a large model, and belongs to the technical field of electric digital data processing, and the method comprises the following steps: collecting resource performance data and lexical element quantity; inputting the performance data and the number of lexical elements into a second large model to obtain predicted performance data and the number of lexical elements; and according to the predicted performance data, the predicted lexical element number and a preset strategy, matching capacity expansion and contraction actions. According to the method, the number of lexical elements and performance data are predicted according to historical data, and the capacity expansion and shrinkage action of a large model is adjusted according to a prediction result, so that the condition of excessive configuration or insufficient resources can be avoided, the overall resource utilization efficiency is improved, and the operation cost is reduced; the method can better adapt to changes under the conditions of high concurrency and burst load, and enhances the elasticity and reliability of large model services.

Description

Technical Field

[0001] The present invention relates to the technical field of electrical digital data processing, and in particular to a method and system for scaling capacity based on a large model. Background Art

[0002] In large-scale language model (big model) services, automatic scaling is the core link to ensure system stability and performance. Existing scaling solutions usually rely on the usage of resource indicators such as CPU, memory, and network I / O to predict and adjust resource configuration. However, there are some basic computing load factors in the use of big models, and this scaling solution does not directly target and consider the specific load of model processing.

[0003] When the number of tokens increases dramatically, resources are insufficiently allocated, which in turn affects service performance. Load prediction methods based on historical data and resource usage may not accurately predict load fluctuations caused by changes in the number of tokens. Existing scaling solutions cannot adapt to load changes in large model services. Summary of the invention

[0004] In view of the above-mentioned technical problems existing in the prior art, the present invention provides a large-model-based scaling method and system, which performs scaling configuration of resources by monitoring the token quantity.

[0005] The present invention discloses a large model-based scaling method, comprising the following steps: collecting resource performance data and word number; inputting the performance data and word number into a second large model to obtain predicted performance data and word number; matching scaling actions according to the predicted performance data and word number, and a preset strategy.

[0006] Preferably, the method for training the second largest model includes: Collect historical data on resource performance and word number of the first model to obtain a performance data set and a corresponding word data set; Constructing a training set according to the performance data set and the word meta data set; Through the training set, the first large model is fine-tuned to obtain the second large model.

[0007] Preferably, the method for constructing the data set labels in the training set includes: Taking the time domain value interval T as a sliding window, constructing a data group according to historical data, the data group includes a first data group and a second data group, and the time series of the sliding window of the second data group is the next time series of the sliding window of the first data group; The second data group is used as the true label of the first data group, and the mean value of the index of the second data group is used as the true value or label value of the corresponding index of the first data group.

[0008] Preferably, the method for converting a data set into a text set is: Obtaining a first template of context text, and generating context text by using the first template and a first data group; Obtain a question template, and generate a question text according to the question template and the time series of the second data group; An answer template is obtained, and an answer text is generated according to the answer template and the second data set.

[0009] Preferably, the data group includes a first data group, a second data group, a third data group and a fourth data group which are sequentially increased in time series, and the method for converting the data group into a text group includes: Obtaining a first template of context text, and generating context text through the first template, the first data group, the second data group, the third data group and their averages; Obtaining a question template, and generating a question text according to the question template and the time series of the fourth data group; An answer template is obtained, and an answer text is generated according to the answer template and the fourth data group.

[0010] Preferably, a text training set is constructed by using the context text, question text and answer text; Fine-tune the first large model using the text training set to obtain a second large model; The second largest model is combined with the first largest model to obtain a third largest model, and the third model is used to output predicted performance data and word unit quantity based on the input performance data and word unit data.

[0011] Preferably, the preset strategy includes: If the predicted performance index is greater than a first threshold and the number of word units is less than a second threshold, increasing the resource of the performance index; If the predicted performance index is less than the first threshold and the number of word units is greater than the second threshold, the number of instances of the second largest model is increased; If the predicted performance index is greater than a first threshold and the number of word units is greater than a second threshold, the resource of the performance index is increased and the number of instances of the second largest model is increased.

[0012] Preferably, the preset strategy also includes: If the predicted performance index is less than the third threshold and the number of word units is less than the second threshold, reducing the resource of the performance index; If the predicted performance index is less than the third threshold and the number of word units is less than the fourth threshold, then the resource of the performance index is reduced and the number of instances of the second largest model is reduced. The fourth threshold is smaller than the second threshold, and the third threshold is smaller than the first threshold.

[0013] The present invention also provides a system for implementing the above method, including a collection module, a prediction module and a decision module, wherein the collection module is used to collect performance data and the number of words; the prediction module is used to input the performance data and the number of words into a second large model to obtain predicted performance data and the number of words; the decision module is used to match the expansion and contraction actions according to the predicted performance data and the number of words, and the preset strategy.

[0014] Preferably, the system further comprises a training module, The training module is used to construct a training set according to the performance data and the word metadata; and to fine-tune the first large model through the training set to obtain the second large model.

[0015] Compared with the prior art, the beneficial effects of the present invention are: predicting the number of words and performance data based on historical data, and adjusting the expansion and contraction actions of the large model according to the prediction results, can avoid over-configuration or insufficient resources, thereby improving overall resource utilization efficiency and reducing operating costs; it can better adapt to changes under high concurrency and burst load conditions, and enhance the elasticity and reliability of large model services. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flow chart of the method for scaling up and down based on a large model of the present invention; Figure 2 It is a system logic block diagram of the present invention. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] The present invention is further described in detail below in conjunction with the accompanying drawings: Overview: Large Language Model (LLM), referred to as large model for short, is a natural language processing model with large-scale parameters and computing power, such as OpenAI's GPT-3 model and ChatGPT model.

[0019] In natural language processing, a word-token refers to the basic unit into which the input text is decomposed, which is understood as the smallest semantic unit. The smallest semantic unit can be a word, a character, or a subword fragment. The number of tokens input by the user indicates the complexity and scale of the model's text processing. More tokens mean that the model needs to process more text information, which usually increases the computational burden and memory usage. The expected response time may become longer, and the computational cost will also increase. A large number of tokens may also affect the generation quality of the model, especially in long texts, where it may be more difficult for the model to maintain contextual consistency. The number of tokens processed is directly related to the computational load. More tokens mean more computing resources are required. Therefore, effectively predicting and monitoring the number of tokens can not only more accurately reflect computing needs, but also optimize resource allocation, thereby improving the system's responsiveness and overall efficiency.

[0020] Especially in high-concurrency situations, the change in the number of tokens may quickly exceed expectations, causing the existing scaling configuration to be unable to adapt, resulting in excessive or insufficient resource allocation, thus affecting cost-effectiveness and service quality.

[0021] To solve this technical problem, the first aspect of the present invention provides a large model-based expansion and contraction method, such as Figure 1 As shown, the following steps are included: Step 101: Collect historical data on resource performance and token quantity of the first large model instance to obtain a performance data set and a corresponding token data set.

[0022] Step 102: construct a training set based on the performance data set and the word-meta data set.

[0023] Step 103: Fine-tune the first largest model using the training set to obtain the second largest model.

[0024] Step 104: Collect performance data and word count.

[0025] Step 105: Input the performance data and the number of word units into the second largest model to obtain the predicted performance data and the number of word units.

[0026] The second largest model is used to predict the performance data and number of tokens at the next moment / time period.

[0027] Step 106: Match the expansion and contraction actions according to the predicted performance data and word quantity, and the preset strategy.

[0028] By predicting the number of words and performance data based on historical data and adjusting the expansion and contraction of large models based on the prediction results, over-configuration or insufficient resources can be avoided, thereby improving overall resource utilization efficiency and reducing operating costs; it can better adapt to changes under high concurrency and sudden load conditions, and enhance the elasticity and reliability of large model services.

[0029] In a specific embodiment, the method for scaling includes the following steps: Step 201: Collect historical data.

[0030] You can use the Prometheus monitoring system to collect and record historical data on resource performance and obtain a data set. The key indicator data that needs to be collected are: CPU utilization, memory usage, network bandwidth utilization, disk I / O, and GPU usage. After determining the indicator data, use Grafaha to create a panel to better display the data change trend in the form of icons, which is convenient for subsequent analysis and improvement.

[0031] For the word count of the user input in the big model, the word count function can be added after the big model instance is modified accordingly, which facilitates the collection of word metadata. Different big models have different word segmentation mechanisms. The word metadata counted by the big model and its instances is more accurate and reliable.

[0032] Step 202: Historical data preprocessing.

[0033] After collecting the raw data, a series of cleaning and preprocessing work is required on the data, including removing abnormal noise data, filling missing values ​​and other operations to ensure the integrity and accuracy of the data. These preprocessing steps are also crucial for subsequent data modeling and predictive analysis.

[0034] Step 203: Divide the historical data into multiple data subsets.

[0035] The data set of historical data is divided into multiple data subsets, wherein the data subsets include: a training set, a validation set and a test set.

[0036] After data cleaning, the data has met the basic specifications. In order to better reflect the dynamic characteristics of time series data, the historical data can also be divided after preprocessing: A data set is constructed with a time domain value interval T as a sliding window, and the data set includes basic performance data and basic word metadata, such as T is 0.5-2 hours, and more specifically 1 hour. Three sliding windows can also be used to divide the historical data into a training set, a validation set, and a test set according to the time sequence.

[0037] The data group includes a first data group and a second data group, the time series of the sliding window of the second data group is the next time series of the sliding window of the first data group; the second data group is used as the true label of the first data group.

[0038] Among them, in each divided data set, T n The data set of the sliding window is the time series T n Historical data, T n+1 The sliding window is the time series T n The predicted value is used as the true label. n The time domain value interval of is represented as: For example, the first data set of the 1-point sliding window represents the performance data and word quantity between 1:00-2:00; the next time series is the second data set corresponding to the 2-point sliding window.

[0039] Step 204: Convert the data group of the data subset into a text group to obtain a text training set, a text verification set, and a text test set.

[0040] Traditional digital and structured historical data does not contain any meaning or semantic related information, and the structured data group cannot be directly passed into the language model in order to convert the data into text efficiently and quickly.

[0041] The data group can be converted into a text group based on the text generator (Data-to-Text Generation): Step 401: Obtain a first template of context text, and generate context text through the first template and a first data group, wherein the text of the first template includes the first data group and its mean.

[0042] Step 402: Obtain a question template, and generate a question text according to the question template and the time series of the first data group, wherein the question template is used to ask a prediction value of the next time series.

[0043] Step 403: Obtain an answer template, and generate an answer text according to the answer template and the second data group of the next time series, wherein the answer template is used to answer the question text through the predicted value of the next time series.

[0044] Step 404: Construct a text training set, a text verification set, and a text test set through the context text, the question text, and the answer text. Construct a text training set through the context text, the question text, and the answer text; Finally, the specific content of the Data-to-Text Generation generator and the generated data can be determined. The specific examples are shown in Table 1.

[0045] Table 1

[0046] Among them, the time series of the first data group, the second data group, the third data group and the fourth data group increase in sequence, and the method of converting the data group into a text group includes: obtaining a first template of context text, and generating context text through the first template, the first data group, the second data group, the third data group and their mean; obtaining a question template, and generating a question text according to the question template and the time series of the fourth data group; obtaining an answer template, and generating an answer text according to the answer template and the fourth data group.

[0047] Step 205: Fine-tune the first large model using the text training set.

[0048] The corresponding large model can be adopted according to the service scenario, and can be selected and downloaded from some libraries of open source libraries and platforms such as Hugging Face. The fine-tuning framework can also be selected according to the needs of training, such as Alignment Handbook, Firefly, LLaMA-Factory, etc. In a specific example, the LLaMA-Factory framework is selected to fine-tune and deploy the large model. LLaMA-Factory has faster instruction fine-tuning, and is more perfect in the pre-training and reinforcement learning process. At the same time, various training methods and deployments have been added in configuration.

[0049] When fine-tuning the model, you can use the LORA method for fine-tuning training and set some necessary hyperparameters during training. LLaMA-Factory also already contains some basic fine-tuning examples. By passing in the corresponding training set, you can complete the fine-tuning and obtain the second largest model.

[0050] Given a pre-trained model, fine-tuning (Fine Tune) based on the basic model can save a lot of computing resources and computing time, improve computing efficiency, and even improve accuracy compared to training from scratch (Training a model from scatch).

[0051] Optionally, the present invention may also merge the final trained second largest model with the basic first largest model as the final third largest model.

[0052] After completing the model training, you can complete some basic performance tests on the large model. Because the training data set and the prediction results are some specific structured data values, you can use the text test set to evaluate the performance of the large model. The specific evaluation method is: first use the text test set to pass the second largest model / third largest model to obtain the feedback prediction value, and calculate the error between the prediction value and the true value, calculate the mean square error, and obtain the performance index of the model.

[0053] If the performance indicators are not satisfactory, you can adjust the values ​​of the hyperparameters or modify the model generated by the text generator and retrain it to get a more satisfactory performance indicator.

[0054] Eventually, the model can be deployed to the production environment to predict the results of scheduled automatic scaling of instances.

[0055] Step 206: Based on the decision maker and the prediction result, the scaling action is matched.

[0056] Performance data can be in the form of the following metrics: CPU usage, memory usage, disk IO, network IO, and GPU usage.

[0057] CPU usage: usually related to the inference speed and concurrent processing capability of the model. For large language models, CPU resources are very important for certain operations (such as pre-processing and model inference). Memory usage: related to the size of the model and the required context window. Large language models usually require a lot of memory to store parameters and process input data. The processing of each token requires a certain amount of memory space. If the memory usage is too high, it may cause frequent garbage collection or even Out of Memory (OOM) errors. Disk IO (also known as disk reading and writing, disk I / O) is related to data reading and writing operations, loading model parameters or storing training data. If disk IO becomes a bottleneck, it may affect the data loading speed and model startup time. Network IO is related to the distributed deployment and data transmission of the model. If the model is deployed on multiple nodes, network IO will affect data synchronization and request response time between nodes. Network IO mainly involves data conversion between user space and kernel space. GPU usage is closely related to the computing performance and training efficiency of the model. Large language models usually require a lot of GPU computing resources to accelerate the inference and training process. High GPU usage indicates that GPU resources are being fully utilized.

[0058] The number of tokens directly affects the inference load and computing resource requirements of large models. Processing more tokens requires more computing and memory resources, and requests for long text or a high number of tokens may result in performance degradation. Because resource usage is vertically scaled, and the number of tokens is related to instances that use horizontal scaling, a load optimization strategy is designed to comprehensively consider scaling based on predicted resource performance indicators and the number of tokens.

[0059] Specifically, when the performance indicators predicted by the model at the next moment, such as CPU, memory and other resource utilization, reach more than 85%, you can increase resource configuration to optimize computing performance. When the tokens that users may enter at the next moment predicted by the model are high, you can increase the number of instances to share the load and improve concurrent processing capabilities. The specific load optimizer is as follows: ; Wherein, target represents the matched scaling action. More specifically: if a certain predicted performance indicator is greater than the first threshold, such as the CPU usage is greater than or equal to 85%, and the number of tokens is less than the second threshold, then the resources of the performance indicator are increased; if a certain performance indicator is less than the first threshold, and the number of tokens is greater than the second threshold, then the number of instances of the large model is increased; if the performance indicator is greater than the first threshold, and the number of tokens is greater than the second threshold, then the resources of the performance indicator are increased, and the number of instances of the large model is increased; if the performance indicator is less than the third threshold, such as the CPU usage is less than 30%, and the number of tokens is less than the second threshold, then the resources of the performance indicator are reduced; if the performance indicator is less than the third threshold, and the number of tokens is less than the fourth threshold, then the resources of the performance indicator are reduced, and the number of instances of the large model is reduced; if the above strategy is not met, the current resources and number of instances are maintained. Wherein, the fourth threshold is less than the second threshold, and the third threshold is less than the first threshold. The matching strategy of the scaling action is not limited to this, and those skilled in the art can configure the matching relationship between performance indicators, tokens and scaling according to the actual usage scenario.

[0060] Finally, according to the matched scaling actions, the number of instances or resource configurations are adjusted using the relevant Kubernetes APIs to meet business needs. The instance of the large model can be the instance of the second largest model or the instance of the third largest model, which can be selected based on business needs and model performance.

[0061] The present invention collects historical performance data and token data, and constructs a data set using a sliding window in a time series manner; based on data set preprocessing and division, the historical structured data is converted into text data through a data text generator, and the collected data is converted into a data set format of a large model to facilitate the training of the large model; the large model is fine-tuned using the prepared data set to obtain a basic prediction model; a scaling decision maker is constructed to use historical data to predict the resources at the next moment and perform automatic scaling.

[0062] Automatically scale up and down the large model based on token prediction, and adjust computing resources and the number of instances by predicting the number of tokens in real time, thereby improving the efficiency and response speed of the system. It can effectively foresee load changes, optimize resource allocation, reduce latency, and reduce costs, while ensuring that the model maintains stable performance when processing tasks of different scales.

[0063] The second aspect of the present invention also provides a system for implementing the above-mentioned expansion and contraction method, such as Figure 2 As shown, it includes a collection module 1, a prediction module 2 and a decision module 3. The acquisition module 1 is used to collect performance data and word number; The prediction module 2 is used to input the performance data and the number of word units into the second large model to obtain the predicted performance data and the number of word units; The decision module 3 is used to match the expansion and contraction actions according to the predicted performance data and word quantity, and the preset strategy.

[0064] The system further comprises a training module 4, which is used to construct a training set according to the performance data and the word metadata; and to fine-tune the first large model through the training set to obtain the second large model.

[0065] The large model of the present invention can be deployed on a big data platform or a cloud platform, and data collection and instance management are performed based on the platform. The cloud platform includes at least one cluster, but in actual application, the number of clusters can be arranged according to the service scale. The number of node groups in the cluster can also be set according to actual needs. The large model in the present invention is actually deployed on the node group.

[0066] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for scaling up and down based on a large model, characterized in that: The following steps are involved: Collect resource performance data and word counts; Input the performance data and the number of word units into the second largest model to obtain predicted performance data and the number of word units; Match the scaling actions based on the predicted performance data, number of words, and preset strategies.

2. The method for capacity expansion and contraction according to claim 1, characterized in that: The training method of the second largest model includes: Collect historical data on resource performance and word number of the first model to obtain a performance data set and a corresponding word data set; Constructing a training set according to the performance data set and the word meta data set; Through the training set, the first large model is fine-tuned to obtain the second large model.

3. The method for capacity expansion and contraction according to claim 2, characterized in that: The methods for constructing data group labels in the training set include: Taking the time domain value interval as a sliding window, constructing a data group according to historical data, the data group includes a first data group and a second data group, and the time series of the sliding window of the second data group is the next time series of the sliding window of the first data group; The second data group is used as the true label of the first data group, or the mean value of the index of the second data group is used as the true value or label value of the corresponding index of the first data group.

4. The method for capacity expansion and contraction according to claim 3, characterized in that: Also included are methods for converting data arrays to text arrays: Obtaining a first template of context text, and generating context text by using the first template and a first data group; Obtain a question template, and generate a question text according to the question template and the time series of the second data group; An answer template is obtained, and an answer text is generated according to the answer template and the second data set.

5. The method for capacity expansion and contraction according to claim 4, characterized in that: The time series of the first data group, the second data group, the third data group and the fourth data group are increased in sequence, and the method for converting the data group into a text group includes: Obtaining a first template of context text, and generating context text through the first template, the first data group, the second data group, the third data group and their averages; Obtaining a question template, and generating a question text according to the question template and the time series of the fourth data group; An answer template is obtained, and an answer text is generated according to the answer template and the fourth data group.

6. The method for expanding and contracting capacity according to claim 4 or 5, characterized in that: Constructing a text training set through the context text, question text and answer text; Fine-tune the first large model using the text training set to obtain a second large model; The second largest model is combined with the first largest model to obtain a third largest model, and the third model is used to output predicted performance data and word unit quantity based on the input performance data and word unit data.

7. The method for capacity expansion and contraction according to claim 1, characterized in that: The preset strategies include: If the predicted performance index is greater than a first threshold and the number of word units is less than a second threshold, increasing the resource of the performance index; If the predicted performance index is less than the first threshold and the number of word units is greater than the second threshold, the number of instances of the second largest model is increased; If the predicted performance index is greater than a first threshold and the number of word units is greater than a second threshold, the resource of the performance index is increased and the number of instances of the second largest model is increased.

8. The method for capacity expansion and contraction according to claim 7, characterized in that: The preset strategy also includes: If the predicted performance index is less than the third threshold and the number of word units is less than the second threshold, reducing the resource of the performance index; If the predicted performance index is less than the third threshold and the number of word units is less than the fourth threshold, then the resource of the performance index is reduced and the number of instances of the second largest model is reduced. The fourth threshold is smaller than the second threshold, and the third threshold is smaller than the first threshold.

9. A system, characterized in that: For implementing the capacity expansion and contraction method according to any one of claims 1 to 8, the system comprises a collection module, a prediction module and a decision module. The collection module is used to collect performance data and word number; The prediction module is used to input the performance data and the number of word units into the second largest model to obtain the predicted performance data and the number of word units; The decision module is used to match the expansion and contraction actions according to the predicted performance data and word quantity, and the preset strategy.

10. The system according to claim 9, characterized in that Also includes training modules, The training module is used to construct a training set according to the performance data and the word metadata; and to fine-tune the first large model through the training set to obtain the second large model.

Images