Dynamic resource optimization method and system based on large model incremental learning

Abstract

The present invention relates to the field of big data processing technology, and in particular to a dynamic resource optimization method and system based on incremental learning of a large model, comprising: updating a model using training data based on a preset training cycle; determining whether the operating status of the model is qualified, and when determining that the operating status of the model is abnormal, adjusting the parameters of each server based on the actual processing time of the to-be-processed data of a single slice, including adjusting the number of servers to a corresponding value, adjusting the network bandwidth of the server corresponding to the to-be-processed data of a single slice to a corresponding value, or adjusting the number of batches of training data selected for updating the model to a corresponding value; determining that the operating status of the model is qualified, and continuously using the current model to complete the allocation of resources. The present invention continuously updates the model according to real-time data, uses a dynamically adjusted resource allocation scheme, and improves resource allocation efficiency.

Description

Technical Field

[0001] The present invention relates to the field of big data processing technology, and in particular to a dynamic resource optimization method and system based on large model incremental learning. Background Art

[0002] In today's digital age, with the rapid development of technologies like the Internet and the Internet of Things, data volumes are exploding. Businesses and organizations are facing increasing demands for data processing efficiency and resource utilization, including transaction data, sensor data, and user behavior data.

[0003] Traditional resource allocation methods are typically based on static rules or experience, such as allocating data to different servers in fixed proportions. However, this approach cannot adapt to dynamic changes in data volume and characteristics, and can easily lead to some servers being overloaded while others remain idle, reducing overall data processing efficiency and increasing operating costs.

[0004] At the same time, with the development of deep learning technology, large models have demonstrated powerful capabilities in data processing and prediction. However, the performance of large models gradually declines as data distribution changes. Therefore, how to use large models for dynamic resource optimization to improve resource utilization efficiency and data processing performance has become a hot research topic.

[0005] Chinese patent application publication number: CN116614396A, discloses a cloud server resource allocation method, including: generating multiple groups of resource pre-allocation information in response to resources requested by users; calculating the supply cost information corresponding to the resource pre-allocation information based on the specification parameters, the quantity parameters and the available zone parameters; determining the user's evaluation information by using the resource activation record data of the user's target resources; determining the resource allocation information in the multiple groups of resource pre-allocation information based on the supply cost information and the evaluation information, and allocating cloud server resources to the user according to the resource allocation information; It can be seen that the above technical solution has the following problems: it does not consider dynamically adjusting resource allocation according to the difference between the actual processing time and the predicted processing time of the data to be processed in each slice, which affects the resource allocation efficiency. Summary of the Invention

[0006] To this end, the present invention provides a dynamic resource optimization method and system based on large-model incremental learning, which is used to overcome the problem in the prior art that resource allocation is not dynamically adjusted according to the difference between the actual processing time and the predicted processing time of the data to be processed in each slice, thereby affecting the efficiency of resource allocation.

[0007] In one aspect, the present invention provides a dynamic resource optimization method based on large model incremental learning, comprising:

[0008] The amount of data received by each server, the amount of data processed, and the corresponding actual processing time are divided into several batches of training data based on the data volume;

[0009] Using the training data based on a preset training cycle, the model for predicting the predicted processing time of the data to be processed for each slice is updated;

[0010] Get the predicted processing time of each slice of data to be processed as predicted by the model;

[0011] Obtain the actual processing time of each server for processing the pending data of each slice;

[0012] Determine the predicted evaluation value of the data to be processed for each slice based on the predicted processing time and the actual processing time;

[0013] The model's operating status is determined based on each prediction evaluation value, including:

[0014] When the model is judged to be operating abnormally, the parameters of each server are adjusted based on the actual processing time of the data to be processed of a single slice, including adjusting the number of servers to a corresponding value, adjusting the network bandwidth of the server corresponding to the data to be processed of a single slice to a corresponding value, or adjusting the number of batches of training data selected for updating the model to a corresponding value;

[0015] When the operating status of the model is determined to be qualified based on each prediction evaluation value one by one, the current model is continuously used to complete the allocation of resources.

[0016] Furthermore, for the data to be processed of a single slice, a predicted evaluation value is determined based on the predicted processing time and the actual processing time, and a process of determining whether the operation status of the model is qualified based on the predicted evaluation value includes:

[0017] The ratio of the calculated actual processing time to the predicted processing time is recorded as the predicted evaluation value;

[0018] If the predicted evaluation value is less than or equal to the first preset evaluation value, the operation status of the model is determined to be qualified;

[0019] If the predicted evaluation value is less than or equal to the second preset evaluation value and greater than the first preset evaluation value, then the operating status of the model is determined to be qualified in combination with the historical call frequency of the to-be-processed data of the single slice;

[0020] If the predicted evaluation value is greater than the second preset evaluation value, the operation status of the model is determined to be abnormal, and the parameters of the corresponding server are adjusted based on the actual processing time of the to-be-processed data of a single slice.

[0021] Furthermore, the historical call frequency of the data to be processed for a single slice is combined to determine whether the model is running properly, including:

[0022] If the historical call frequency is less than or equal to the preset historical call frequency, the model is judged to be in good condition;

[0023] If the historical call frequency is greater than the preset historical call frequency, the model's operating status is determined to be abnormal, and the parameters of the corresponding server are adjusted based on the actual processing time of the to-be-processed data of a single slice.

[0024] Furthermore, the process of adjusting the parameters of the corresponding server based on the actual processing time of the to-be-processed data of a single slice includes:

[0025] If the actual processing time is less than or equal to the first preset actual processing time, determining a predicted fluctuation parameter based on the predicted evaluation value of the to-be-processed data of each slice, and adjusting the parameters of the corresponding server based on the predicted fluctuation parameter;

[0026] If the actual processing time is less than or equal to the second preset actual processing time and greater than the first preset actual processing time, the number of servers is adjusted to a corresponding value based on the average evaluation value of the predicted evaluation values ​​of the to-be-processed data of each slice;

[0027] If the actual processing time is longer than the second preset actual processing time, the data repetition rate is determined based on the data content of the to-be-processed data of a single slice, and the parameters of the corresponding server are adjusted based on the data repetition rate.

[0028] Furthermore, the process of adjusting the parameters of the corresponding server based on the predicted fluctuation parameter includes:

[0029] The variance of the predicted evaluation value of the data to be processed of each slice is recorded as the predicted fluctuation parameter;

[0030] If the predicted fluctuation parameter is less than or equal to the preset predicted fluctuation parameter, the network bandwidth of the server corresponding to the to-be-processed data of a single slice is adjusted to the corresponding value based on the actual processing time;

[0031] If the predicted fluctuation parameter is greater than the preset predicted fluctuation parameter, the batch selection number of the training data selected for updating the model is adjusted to a corresponding value based on the predicted fluctuation parameter.

[0032] Furthermore, the number of servers is adjusted to a corresponding value based on the average evaluation value, wherein,

[0033] The increase in the number of servers is proportional to the average evaluation value.

[0034] Furthermore, the process of determining a data repetition rate based on the data content of the to-be-processed data of a single slice and adjusting the parameters of each server based on the data repetition rate includes:

[0035] Determine the data sending end corresponding to the data to be processed of a single slice;

[0036] Get the data sent by the data sender in history;

[0037] Calculate the repetition rate between each sent data;

[0038] Solve the average value of each repetition rate to obtain the data repetition rate;

[0039] If the data repetition rate is less than or equal to the preset data repetition rate, the number of servers is adjusted to a corresponding value based on the average evaluation value;

[0040] If the data repetition rate is greater than the preset data repetition rate, the single data sending end is marked as an orderly sending end.

[0041] Furthermore, the network bandwidth of the server corresponding to the data to be processed of a single slice is adjusted to a corresponding value based on the actual processing time, wherein,

[0042] The increase in network bandwidth is proportional to the actual processing time.

[0043] Furthermore, the number of batches of training data selected for model updating is adjusted to a corresponding value based on the predicted fluctuation parameter, wherein:

[0044] The increase in the number of batches selected is proportional to the predicted fluctuation parameter.

[0045] On the other hand, the present invention also provides a dynamic resource optimization system based on a dynamic resource optimization method of large model incremental learning, comprising:

[0046] A data output module, comprising a plurality of data sending terminals for sending data to be processed;

[0047] a model processing module connected to the data output module, configured to divide the data sent by each data sending end into a plurality of slices of data to be processed, and determine the optimal resource allocation scheme for each server to process the data to be processed and the processing time of each server to process the data to be processed;

[0048] A data receiving module, which is connected to the data output module and the model processing module respectively, and includes a plurality of servers for receiving and processing the data to be processed of each slice sent by each data sending end according to the optimal resource allocation solution;

[0049] A data statistics module, connected to the data receiving module, for counting the actual processing time of each server for processing the data to be processed in each slice;

[0050] A data partitioning module, which is connected to the model processing module, the data receiving module and the data statistics module respectively, and is used to divide the amount of data received by the server, the amount of data processed and the corresponding actual processing time into several batches of training data according to the amount of data;

[0051] A model training module, which is connected to the data partitioning module and the model processing module respectively, and is used to select several batches of training data to update the model;

[0052] A data analysis module is respectively connected to the model processing module, the data statistics module, the data receiving module and the model training module, and is used to determine a predicted evaluation value based on the predicted processing time and the actual processing time, judge whether the operating status of the model is qualified based on the predicted evaluation value, and when it is judged that the operating status of the model is abnormal, adjust the parameters of the corresponding server based on the actual processing time of the to-be-processed data of a single slice, including adjusting the number of servers to a corresponding value, adjusting the network bandwidth of the server corresponding to the to-be-processed data of a single slice to a corresponding value, or adjusting the number of batches of training data selected for updating the model to a corresponding value.

[0053] Compared with the prior art, the present invention has the following advantages: it updates the model using training data based on a preset training cycle; determines whether the model's operating status is qualified; and when it is determined that the model's operating status is abnormal, adjusts the parameters of each server based on the actual processing time of the to-be-processed data of a single slice, including adjusting the number of servers to a corresponding value, adjusting the network bandwidth of the server corresponding to the to-be-processed data of a single slice to a corresponding value, or adjusting the number of batches of training data selected for updating the model to a corresponding value; determines that the model's operating status is qualified, and continuously uses the current model to complete resource allocation. Continuously updating the model based on real-time data and dynamically adjusting the resource allocation plan improves resource allocation efficiency.

[0054] Furthermore, when obtaining the model's predicted processing of each slice of pending data, the slice pending data is a subset of the pending data. Based on the input pending data, the model predicts the processing time for each slice on each server. The ratio of the calculated actual processing time to the predicted processing time is recorded as the prediction evaluation value. The prediction evaluation value reflects the accuracy of the model's prediction; the closer the ratio is to 1, the more accurate the model's prediction. When the prediction evaluation value is less than or equal to a first preset evaluation value, the model's prediction accuracy is high, the model's performance is determined to be satisfactory, and the current model is continued to be used to complete resource allocation. When the prediction evaluation value is less than or equal to a second preset evaluation value and greater than the first preset evaluation value, the model's prediction accuracy has decreased, but it cannot be directly determined that the model is operating abnormally; a judgment based on historical call frequency is required. When the prediction evaluation value is greater than the second preset evaluation value, the model's prediction error is large, the model's performance is determined to be abnormal, and the parameters of each server are adjusted based on the actual processing time of the individual slices of pending data. Dynamically adjusting the parameters and number of servers based on the prediction evaluation value enables adaptive adjustment based on actual conditions, improving the reliability and efficiency of resource allocation.

[0055] Furthermore, based on the historical call frequency evaluation model, the historical call frequency characterizes the frequency of data usage. For data with a higher usage frequency, the model's prediction accuracy is required to be higher, and the processed data needs to be obtained more promptly. If the historical call frequency is less than or equal to the preset historical call frequency, the frequency of use of the data is low. Even if there is a certain error in the model prediction, the impact on the call of the data is small. The model's operating status is determined to be qualified, and the current model is continuously used to complete the allocation of resources. When the historical call frequency is greater than the preset historical call frequency, the frequency of use of the data is high, and the model's prediction error will have a greater impact on the call of the data. In this case, the model's operating status is determined to be abnormal, and the parameters of each server are adjusted based on the actual processing time of a single share of the data to be processed. The analysis standard of the model is determined based on the actual situation of the data to avoid excessive adjustment of the server's operating parameters, resulting in waste of resources, and improving the efficiency of resource allocation.

[0056] Furthermore, based on the actual processing time adjustment strategy, the actual processing time characterizes the deviation amplitude of the actual predicted processing time when the predicted evaluation value is large. When the actual processing time is less than or equal to the first preset actual processing time, the data processing speed is faster. In this case, the predicted fluctuation parameter is determined based on the predicted evaluation value of the data to be processed of each slice. The predicted fluctuation parameter characterizes the error difference of the model on the data to be processed of each slice. When the predicted fluctuation parameter is less than or equal to the preset predicted fluctuation parameter, the model's predictions for the data to be processed of each slice have similar differences. In this case, the predictions are different due to the abnormal network environment. In this case, the network bandwidth is adjusted to ensure stable data transmission and processing. When the predicted fluctuation parameter is greater than the preset predicted fluctuation parameter, in this case, the resource allocation is abnormal due to the abnormality of the model, resulting in different degrees of error in the prediction of the data of each slice. In this case, the number of batches of training data selected for updating the model is increased to improve the update effect of the model and ensure stable operation of the model. When the actual processing time is less than or equal to the second preset actual processing time and greater than the first preset actual processing time, in this case, due to server abnormalities, the sliced ​​data requires a longer processing time to complete data transmission and processing. In this case, the number of servers is adjusted to improve data processing performance. When the actual processing time is greater than the second preset actual processing time, the data processing speed is slow in this case. The data repetition rate is determined based on the data content of a single share of the data to be processed, and the parameters of each server are adjusted based on the data repetition rate. According to indicators such as the model's predicted evaluation value, actual processing time, historical call frequency, and data repetition rate, the server parameters are dynamically adjusted, and adaptive adjustments can be made according to actual conditions to improve the reliability of resource allocation, thereby improving the efficiency of resource allocation.

[0057] Furthermore, when the data repetition rate is less than or equal to the preset data repetition rate, data redundancy is low, and the number of servers is increased to ensure data processing efficiency. When the data repetition rate is greater than the preset data repetition rate, data redundancy is high, and the individual data sender is marked as an orderly sender to facilitate subsequent data processing. The model predicts the resource allocation plan and processing time for the data to be processed, and distributes the data to various servers, avoiding resource waste and overload, and improving overall data processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is a flowchart of the steps of a dynamic resource optimization method based on large model incremental learning according to an embodiment of the present invention;

[0059] Figure 2 This is a module block diagram of a dynamic resource optimization system based on large model incremental learning according to an embodiment of the present invention;

[0060] Figure 3A logic decision diagram for determining whether the operating status of a model is qualified based on the predicted evaluation value according to an embodiment of the present invention;

[0061] Figure 4 This is a logical decision diagram for determining whether the operating status of a model is qualified based on historical call frequency according to an embodiment of the present invention. DETAILED DESCRIPTION

[0062] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are merely used to explain the present invention and are not intended to limit the present invention.

[0063] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0064] It should be noted that, in the description of the present invention, terms such as "up", "down", "left", "right", "inside", and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the accompanying drawings. This is only for the convenience of description and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on the present invention.

[0065] Furthermore, it should be noted that, in the description of the present invention, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed connections, detachable connections, or integral connections; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0066] See also Figure 1 、 Figure 2 、 Figure 3 as well as Figure 4 As shown, they are respectively a flowchart of a dynamic resource optimization method based on large model incremental learning according to an embodiment of the present invention, a module block diagram of a dynamic resource optimization system based on large model incremental learning, a logical decision diagram for determining whether the operating status of a model is qualified based on a predicted evaluation value, and a logical decision diagram for determining whether the operating status of a model is qualified based on a historical call frequency; a dynamic resource optimization method and system based on large model incremental learning according to an embodiment of the present invention, comprising:

[0067] S1: The amount of data received by each server, the amount of data processed, and the corresponding actual processing time are divided into several batches of training data based on the data volume;

[0068] S2, using the training data based on a preset training cycle to update a model for predicting the predicted processing time of the data to be processed for each slice;

[0069] S3, obtaining the predicted processing time of each slice of the data to be processed predicted by the model;

[0070] S4, obtaining the actual processing time of each server for processing the data to be processed in each slice;

[0071] S5, determining a predicted evaluation value of the to-be-processed data of each slice based on the predicted processing time and the actual processing time;

[0072] S6, judging whether the model's operating status is qualified based on each prediction evaluation value one by one, including:

[0073] When the model is judged to be operating abnormally, the parameters of each server are adjusted based on the actual processing time of the data to be processed of a single slice, including adjusting the number of servers to a corresponding value, adjusting the network bandwidth of the server corresponding to the data to be processed of a single slice to a corresponding value, or adjusting the number of batches of training data selected for updating the model to a corresponding value;

[0074] S7, when it is determined that the operating status of the model is qualified according to each prediction evaluation value one by one, the current model is continuously used to complete the allocation of resources.

[0075] Specifically, the process of dividing the amount of data received by each server, the amount of data processed, and the corresponding actual processing time into several batches of training data based on the amount of data includes:

[0076] The amount of data received by a single server, the amount of data processed, and the corresponding actual processing time are packaged into a single piece of data; each piece of data from each server is collected and divided into several batches of training data.

[0077] Specifically, the model is updated using training data based on a preset training cycle. The model's operational status is determined to be acceptable. If the model's operational status is determined to be abnormal, the parameters of each server are adjusted based on the actual processing time of the pending data for a single slice. This includes adjusting the number of servers to a corresponding value, adjusting the network bandwidth of the servers corresponding to the pending data for a single slice to a corresponding value, or adjusting the number of batches of training data selected for updating the model to a corresponding value. If the model's operational status is determined to be acceptable, the current model is continuously used to complete resource allocation. Continuously updating the model based on real-time data and dynamically adjusting the resource allocation plan improves resource allocation efficiency.

[0078] Specifically, there is no restriction on the specific method of obtaining training data. The amount of received data, the amount of processed data, and the corresponding actual processing time can be obtained from the log system of each server. Training data can be collected through the server's built-in monitoring tools such as Linux system sar, iostat, or professional monitoring software such as Zabbix and Prometheus.

[0079] Specifically, there is no limitation on the data volume division. In this embodiment, optionally, the training data can be divided into batches according to the size of the data volume.

[0080] Specifically, the training data is divided into batches so that part of the batch data can be loaded into the memory for model training to avoid insufficient memory and memory overflow errors caused by training a large amount of data at one time. By dividing the data into smaller batches and loading only part of the batch data into the memory for processing each time, the memory usage can be effectively controlled to ensure the stability of the training process.

[0081] Specifically, the model is trained with batches of training data to enable the model to more accurately predict the resource allocation plan for the data to be processed.

[0082] Specifically, for the model, you can choose the recurrent neural network in deep learning and its variants LSTM, GRU, or the model based on the Transformer architecture, and BERT as the basic model to achieve the function of distributing processing sequence data and time series prediction.

[0083] Specifically, model training uses input data including the amount of data received by the server, the amount of data processed, and the actual processing time for each batch of training data. This data is normalized using the Min-Max normalization method, mapping the data to the range [0, 1]. Model training is performed using stochastic gradient descent (SGD) and its optimization algorithms, Adam and Adagrad. During each pre-set training cycle, the model is incrementally trained using new training data to update the model parameters.

[0084] Specifically, for model acquisition, you can use open-source deep learning frameworks such as TensorFlow and PyTorch to build and train the model. After training, save the model as a file for subsequent use.

[0085] Specifically, the model is used to determine the optimal resource allocation solution for each server to process the data to be processed and the processing time of each server to process the data to be processed.

[0086] Specifically, the resource optimization plan includes dividing the data to be processed into several slices, where the amount of data in each slice is different, and determining that the data to be processed in the corresponding slice will be allocated to the corresponding server for processing. The trained model predicts the processing time of each data slice to obtain the predicted processing time.

[0087] Specifically, dynamic resources include several servers for processing data to be processed.

[0088] Specifically, when each server processes each data slice, it records the actual processing time. The recording of the actual processing time can be achieved by adding a time recording code in the server processing program.

[0089] Specifically, after the adjustment of the parameters of each server is completed, whether the operating status of the model is qualified is determined one by one based on each newly acquired prediction evaluation value.

[0090] Specifically, for the data to be processed of a single slice, a predicted evaluation value is determined based on the predicted processing time and the actual processing time, and the process of determining whether the model's operating status is qualified based on the predicted evaluation value includes:

[0091] The ratio of the calculated actual processing time to the predicted processing time is recorded as the predicted evaluation value;

[0092] If the predicted evaluation value is less than or equal to the first preset evaluation value, the operation status of the model is determined to be qualified;

[0093] If the predicted evaluation value is less than or equal to the second preset evaluation value and greater than the first preset evaluation value, then the operating status of the model is determined to be qualified in combination with the historical call frequency of the to-be-processed data of the single slice;

[0094] If the predicted evaluation value is greater than the second preset evaluation value, the operation status of the model is determined to be abnormal, and the parameters of the corresponding server are adjusted based on the actual processing time of the to-be-processed data of a single slice.

[0095] Specifically, the first preset evaluation value is selected within the interval [0.9, 1.1], and the second preset evaluation value is selected within the interval [1.2, 1.3].

[0096] Specifically, when obtaining the model's predictions for each slice of pending data, the slices of pending data are subsets of the pending data. Based on the input pending data, the model predicts the processing time for each slice on each server. The ratio of the calculated actual processing time to the predicted processing time is recorded as the prediction evaluation value. The prediction evaluation value reflects the accuracy of the model's prediction; the closer the ratio is to 1, the more accurate the model's prediction. When the prediction evaluation value is less than or equal to a first preset evaluation value, the model's prediction accuracy is high, the model's performance is determined to be satisfactory, and resource allocation is continued using the current model. When the prediction evaluation value is less than or equal to a second preset evaluation value and greater than the first preset evaluation value, the model's prediction accuracy decreases, but it cannot be directly determined that the model is operating abnormally; a judgment based on historical call frequency is required. When the prediction evaluation value is greater than the second preset evaluation value, the model's prediction error is large, the model's performance is determined to be abnormal, and the parameters of each server are adjusted based on the actual processing time of the individual slices of pending data. Dynamically adjusting the parameters and number of servers based on the prediction evaluation value enables adaptive adjustments based on actual conditions, improving the reliability and efficiency of resource allocation.

[0097] Specifically, the model's operational status is determined based on the historical call frequency of the data to be processed for a single slice, including:

[0098] If the historical call frequency is less than or equal to the preset historical call frequency, the model is judged to be in good condition;

[0099] If the historical call frequency is greater than the preset historical call frequency, the model's operating status is determined to be abnormal, and the parameters of the corresponding server are adjusted based on the actual processing time of the to-be-processed data of a single slice.

[0100] Specifically, the preset historical call frequency is selected within the interval [0.1, 0.3], and the unit is times / day.

[0101] Specifically, the data sending end corresponding to the data to be processed of a single slice is obtained, and the ratio of the number of times the server calls each data sent by the data sending end within the preset analysis time to the preset analysis time is obtained to obtain the historical call frequency.

[0102] Specifically, based on the historical call frequency evaluation model, the historical call frequency characterizes the frequency of data usage. For data with a higher usage frequency, the model's prediction accuracy is required to be higher, and the processed data needs to be obtained more promptly. If the historical call frequency is less than or equal to the preset historical call frequency, the frequency of use of the data is low. Even if there is a certain error in the model prediction, the impact on the call of the data is small. The model's operating status is determined to be qualified, and the current model is continuously used to complete the allocation of resources. When the historical call frequency is greater than the preset historical call frequency, the frequency of use of the data is high, and the model's prediction error will have a greater impact on the call of the data. In this case, the model's operating status is determined to be abnormal, and the parameters of each server are adjusted based on the actual processing time of a single share of the data to be processed. The analysis standard of the model is determined based on the actual situation of the data to avoid excessive adjustment of the server's operating parameters, resulting in waste of resources, and improving the efficiency of resource allocation.

[0103] Specifically, the process of adjusting the parameters of the corresponding server based on the actual processing time of the to-be-processed data of a single slice includes:

[0104] If the actual processing time is less than or equal to the first preset actual processing time, determining a predicted fluctuation parameter based on the predicted evaluation value of the to-be-processed data of each slice, and adjusting the parameters of the corresponding server based on the predicted fluctuation parameter;

[0105] If the actual processing time is less than or equal to the second preset actual processing time and greater than the first preset actual processing time, the number of servers is adjusted to a corresponding value based on the average evaluation value of the predicted evaluation values ​​of the to-be-processed data of each slice;

[0106] If the actual processing time is longer than the second preset actual processing time, the data repetition rate is determined based on the data content of the to-be-processed data of a single slice, and the parameters of the corresponding server are adjusted based on the data repetition rate.

[0107] Specifically, the first preset actual processing time T1 is selected within the interval [10, 20], and the second preset actual processing time T2 is selected within the interval [40, 55], and the unit is seconds.

[0108] Specifically, the process of adjusting the parameters of the corresponding server based on the predicted fluctuation parameters includes:

[0109] The variance of the predicted evaluation value of the data to be processed of each slice is recorded as the predicted fluctuation parameter;

[0110] If the predicted fluctuation parameter is less than or equal to the preset predicted fluctuation parameter, the network bandwidth of the server corresponding to the to-be-processed data of a single slice is adjusted to the corresponding value based on the actual processing time;

[0111] Specifically, the network bandwidth of the server corresponding to the processed data of a single slice is adjusted to the corresponding value based on the actual processing time. The single slice mentioned is the single slice used to determine the predicted evaluation value in the above-mentioned judgment of whether the operating status of the model is qualified based on the predicted evaluation value.

[0112] If the predicted fluctuation parameter is greater than the preset predicted fluctuation parameter, the batch selection number of the training data selected for updating the model is adjusted to a corresponding value based on the predicted fluctuation parameter.

[0113] Specifically, the preset predicted fluctuation parameter Y0 is selected within the interval [0.001, 0.005].

[0114] Specifically, based on the actual processing time adjustment strategy, the actual processing time characterizes the deviation amplitude of the actual predicted processing time when the predicted evaluation value is large. When the actual processing time is less than or equal to the first preset actual processing time, the data processing speed is faster. In this case, the predicted fluctuation parameter is determined based on the predicted evaluation value of the data to be processed of each slice. The predicted fluctuation parameter characterizes the error difference of the model on the data to be processed of each slice. When the predicted fluctuation parameter is less than or equal to the preset predicted fluctuation parameter, the model's predictions for the data to be processed of each slice have similar differences. In this case, the predictions are different due to the abnormal network environment. In this case, the network bandwidth of the server corresponding to the data to be processed of a single slice is adjusted to ensure stable data transmission and processing. When the predicted fluctuation parameter is greater than the preset predicted fluctuation parameter, in this case, the resource allocation is abnormal due to the abnormality of the model, resulting in different degrees of error in the prediction of the data of each slice. In this case, the number of batches of training data selected for updating the model is increased to improve the update effect of the model and ensure stable operation of the model. When the actual processing time is less than or equal to the second preset actual processing time and greater than the first preset actual processing time, in this case, due to server abnormalities, the sliced ​​data requires a longer processing time to complete data transmission and processing. In this case, the number of servers is adjusted to improve data processing performance. When the actual processing time is greater than the second preset actual processing time, the data processing speed is slow in this case. The data repetition rate is determined based on the data content of a single share of the data to be processed, and the parameters of each server are adjusted based on the data repetition rate. According to indicators such as the model's predicted evaluation value, actual processing time, historical call frequency, and data repetition rate, the server parameters are dynamically adjusted, and adaptive adjustments can be made according to actual conditions to improve the reliability of resource allocation, thereby improving the efficiency of resource allocation.

[0115] Specifically, the number of servers is adjusted to a corresponding value based on the average evaluation value, where

[0116] The increase in the number of servers is proportional to the average evaluation value.

[0117] In this embodiment, optionally,

[0118] Comparing the average evaluation value with a first preset average evaluation value and a second preset average evaluation value;

[0119] If the average evaluation value is less than or equal to the first preset average evaluation value, the number of servers is adjusted to 1.1 times the initial number of servers;

[0120] If the average evaluation value is less than or equal to the second preset average evaluation value and greater than the first preset average evaluation value, the number of servers is adjusted to 1.2 times the initial number of servers;

[0121] If the average evaluation value is greater than the second preset average evaluation value, the number of servers is adjusted to 1.25 times the initial number of servers;

[0122] The first preset average evaluation value is 1.05, and the second preset average evaluation value is 1.15.

[0123] Specifically, the process of determining the data repetition rate based on the data content of the to-be-processed data of a single slice and adjusting the parameters of each server based on the data repetition rate includes:

[0124] Determine the data sending end corresponding to the data to be processed of a single slice;

[0125] Get the data sent by the data sender in history;

[0126] Calculate the repetition rate between each sent data;

[0127] Solve the average value of each repetition rate to obtain the data repetition rate;

[0128] If the data repetition rate is less than or equal to the preset data repetition rate, the number of servers is adjusted to a corresponding value based on the average evaluation value;

[0129] If the data repetition rate is greater than the preset data repetition rate, the single data sending end is marked as an orderly sending end.

[0130] Specifically, the preset data repetition rate is selected within the interval [0.4, 0.6].

[0131] Specifically, for the data sent twice, the ratio of the calculated amount of repeated data to the total amount of data sent twice is recorded as the repetition rate between the data sent twice.

[0132] Specifically, when a single server receives data to be processed sent by an order sending end, it identifies the difference areas between the data to be processed and the template sent by the order sending end, and only performs analysis and processing on the difference areas.

[0133] Specifically, the specific method of analyzing and processing the difference area is not limited and may include data standardization and inputting the standardized data into a pre-stored program for analysis, which will not be described in detail.

[0134] Specifically, the difference area is an area where the identified data to be processed is different from the sent template.

[0135] Specifically, the sending template is determined as follows:

[0136] Calculate the sum of the repetition rates of the data sent once in history and the data sent each time by the sender to obtain a reference repetition rate for the data sent once in history;

[0137] The data sent once and corresponding to the maximum value of each reference repetition rate of the selected data sent once in history is determined as the sending template.

[0138] Specifically, when the data repetition rate is less than or equal to the preset data repetition rate, data redundancy is low, and the number of servers is increased to ensure data processing efficiency. When the data repetition rate is greater than the preset data repetition rate, data redundancy is high, and individual data senders are marked as orderly senders to facilitate subsequent data processing. The model predicts the resource allocation plan and processing time for the data to be processed, and distributes the data to various servers, avoiding resource waste and overload, and improving overall data processing efficiency.

[0139] Specifically, the network bandwidth of the server corresponding to the data to be processed of a single slice is adjusted to the corresponding value based on the actual processing time, where:

[0140] The increase in network bandwidth is proportional to the actual processing time.

[0141] In this embodiment, optionally,

[0142] Comparing the actual processing time with the first time comparison threshold and the second time comparison threshold;

[0143] If the actual processing time is less than or equal to the first time comparison threshold, the network bandwidth of the server corresponding to the to-be-processed data of the single slice is adjusted to 1.12 times the initial network bandwidth;

[0144] If the actual processing time is less than or equal to the second time comparison threshold and greater than the first time comparison threshold, the network bandwidth of the server corresponding to the to-be-processed data of the single slice is adjusted to 1.21 times the initial network bandwidth;

[0145] If the actual processing time is longer than the second time comparison threshold, the network bandwidth of the server corresponding to the to-be-processed data of the single slice is adjusted to 1.28 times the initial network bandwidth;

[0146] The first duration comparison threshold is 0.5T1, and the second duration comparison threshold is 0.7T1.

[0147] Specifically, the number of batches of training data for selecting the updated model is adjusted to a corresponding value based on the predicted fluctuation parameter, wherein,

[0148] The increase in the number of batches selected is proportional to the predicted fluctuation parameter.

[0149] In this embodiment, optionally,

[0150] comparing the predicted fluctuation parameter with a first fluctuation comparison threshold and a second fluctuation comparison threshold;

[0151] If the predicted fluctuation parameter is less than or equal to the first fluctuation comparison threshold, the number of batches of training data selected for updating the model is adjusted to 1.15 times the number of the initial batches;

[0152] If the predicted fluctuation parameter is less than or equal to the second fluctuation comparison threshold and greater than the first fluctuation comparison threshold, the number of batches of training data selected for updating the model is adjusted to 1.23 times the number of the initial batches;

[0153] If the predicted fluctuation parameter is greater than the second fluctuation comparison threshold, the number of batches of training data selected for updating the model is adjusted to 1.3 times the number of the initial batches;

[0154] The first fluctuation comparison threshold is 4Y0, and the second fluctuation comparison threshold is 6Y0.

[0155] Specifically, the data output module includes a number of data sending terminals for sending data to be processed;

[0156] a model processing module connected to the data output module, configured to divide the data sent by each data sending end into a plurality of slices of data to be processed, and determine the optimal resource allocation scheme for each server to process the data to be processed and the processing time of each server to process the data to be processed;

[0157] A data receiving module, which is connected to the data output module and the model processing module respectively, and includes a plurality of servers for receiving and processing the data to be processed of each slice sent by each data sending end according to the optimal resource allocation solution;

[0158] A data statistics module, connected to the data receiving module, for counting the actual processing time of each server for processing the data to be processed in each slice;

[0159] A data partitioning module, which is connected to the model processing module, the data receiving module and the data statistics module respectively, and is used to divide the amount of data received by the server, the amount of data processed and the corresponding actual processing time into several batches of training data according to the amount of data;

[0160] A model training module, which is connected to the data partitioning module and the model processing module respectively, and is used to select several batches of training data to update the model;

[0161] A data analysis module is respectively connected to the model processing module, the data statistics module, the data receiving module and the model training module, and is used to determine a predicted evaluation value based on the predicted processing time and the actual processing time, judge whether the operating status of the model is qualified based on the predicted evaluation value, and when it is judged that the operating status of the model is abnormal, adjust the parameters of the corresponding server based on the actual processing time of the to-be-processed data of a single slice, including adjusting the number of servers to a corresponding value, adjusting the network bandwidth of the server corresponding to the to-be-processed data of a single slice to a corresponding value, or adjusting the number of batches of training data selected for updating the model to a corresponding value.

[0162] Specifically, the structure of the data transmitter in the data output module is not limited and can be a variety of data sources, including sensors, business systems, and user terminals. These data transmitters can send unprocessed data to the model processing module and data receiving module via network interfaces such as HTTP and TCP. To ensure the orderly delivery and stable transmission of data, a buffer can be set up at the data transmitter to cache and queue unprocessed data.

[0163] Specifically, there is no limitation on the specific method of dividing the data to be processed into slices. The data can be sliced ​​according to its size, type or business needs. For large data sets, it can be divided according to a fixed amount of data; for time series data, it can be divided according to time windows.

[0164] Specifically, for resource allocation and processing time prediction, a trained model can be used to determine the optimal resource allocation plan for each server processing pending data and the processing time required for each server to process the data. This model can be based on deep learning, such as LSTM or GRU. It is understood that by learning from historical data, it can predict the processing time of different data slices on different servers, thereby achieving optimal resource allocation.

[0165] Specifically, there is no limitation on the specific way in which the server processes the data sent by each data sending end, which may include data classification, data preprocessing, analyzing the data based on a preset program and outputting an analysis structure, and data storage.

[0166] Specifically, the specific structure of the data statistics module is not limited and can be any logical component. It is understood that it can be used to count the actual processing time of each server processing the data of each slice. This can be achieved by adding time recording code to the server processing program, recording the time when data starts to be received and the time when processing is completed. The difference between the two is the actual processing time.

[0167] Specifically, the server processes data sent by each data sender through data classification, dividing the data in the dataset into different categories based on specific rules or characteristics. For telecommunications equipment fault handling data, data can be classified based on characteristics such as equipment type, fault level, and professional type. Data classification can be based on collected information such as operation time, process status, changes in operator roles, early warning feedback, fault level, fault category, site information, equipment information, and data update information. Data preprocessing involves standardizing the data format to ensure data integrity and accuracy. Data is analyzed based on pre-set procedures and the analysis results are output. Early warning analysis is performed on the data, analyzing time, quantity, and management based on set thresholds to determine the warning level and output a warning report. Different servers process data from different senders.

[0168] Specifically, the model training module updates the model by selecting batches of training data from the data partitioning module and loading the current model parameters from a previously trained and saved model file. The model is trained using stochastic gradient descent (SGD) and its optimization algorithms, Adam and Adagrad. During training, the training data is input into the model, and the mean squared error (MSE) between the model output and the actual processing time is calculated. The model parameters are then updated using backpropagation of this error. After training is complete, the updated model parameters are saved to a file for subsequent use.

[0169] Thus far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present invention.

[0170] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that the present invention is susceptible to various modifications and variations. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A dynamic resource optimization method based on large model incremental learning, characterized in that: include: The amount of data received by each server, the amount of data processed, and the corresponding actual processing time are divided into several batches of training data based on the data volume; Using the training data based on a preset training cycle, the model for predicting the predicted processing time of the data to be processed for each slice is updated; Get the predicted processing time of each slice of data to be processed as predicted by the model; Obtain the actual processing time of each server for processing the pending data of each slice; Determine the predicted evaluation value of the data to be processed for each slice based on the predicted processing time and the actual processing time; The model's operating status is determined based on each prediction evaluation value, including: When the model is judged to be operating abnormally, the parameters of each server are adjusted based on the actual processing time of the data to be processed of a single slice, including adjusting the number of servers to a corresponding value, adjusting the network bandwidth of the server corresponding to the data to be processed of a single slice to a corresponding value, or adjusting the number of batches of training data selected for updating the model to a corresponding value; When the operation status of the model is determined to be qualified according to each prediction evaluation value one by one, the current model is continuously used to complete the allocation of resources; For the data to be processed for a single slice, a predicted evaluation value is determined based on the predicted processing time and the actual processing time, and the process of determining whether the model's operating status is qualified based on the predicted evaluation value includes: The ratio of the calculated actual processing time to the predicted processing time is recorded as the predicted evaluation value; If the predicted evaluation value is less than or equal to the first preset evaluation value, the operation status of the model is determined to be qualified; If the predicted evaluation value is less than or equal to the second preset evaluation value and greater than the first preset evaluation value, then the operating status of the model is determined to be qualified in combination with the historical call frequency of the to-be-processed data of the single slice; If the predicted evaluation value is greater than the second preset evaluation value, the operation status of the model is determined to be abnormal, and the parameters of the corresponding server are adjusted based on the actual processing time of the to-be-processed data of a single slice; The model's operational status is determined based on the historical call frequency of the data to be processed for a single slice, including: If the historical call frequency is less than or equal to the preset historical call frequency, the model is judged to be in good condition; If the historical call frequency is greater than the preset historical call frequency, the model's operating status is determined to be abnormal, and the parameters of the corresponding server are adjusted based on the actual processing time of the to-be-processed data of a single slice.

2. The dynamic resource optimization method based on large model incremental learning according to claim 1 is characterized in that: The process of adjusting the corresponding server parameters based on the actual processing time of the pending data of a single slice includes: If the actual processing time is less than or equal to the first preset actual processing time, determining a predicted fluctuation parameter based on the predicted evaluation value of the to-be-processed data of each slice, and adjusting the parameters of the corresponding server based on the predicted fluctuation parameter; If the actual processing time is less than or equal to the second preset actual processing time and greater than the first preset actual processing time, the number of servers is adjusted to a corresponding value based on the average evaluation value of the predicted evaluation values ​​of the to-be-processed data of each slice; If the actual processing time is longer than the second preset actual processing time, the data repetition rate is determined based on the data content of the to-be-processed data of a single slice, and the parameters of the corresponding server are adjusted based on the data repetition rate.

3. The dynamic resource optimization method based on large model incremental learning according to claim 2 is characterized in that: The process of adjusting the parameters of the corresponding server based on the predicted fluctuation parameters includes: The variance of the predicted evaluation value of the data to be processed of each slice is recorded as the predicted fluctuation parameter; If the predicted fluctuation parameter is less than or equal to the preset predicted fluctuation parameter, the network bandwidth of the server corresponding to the to-be-processed data of a single slice is adjusted to the corresponding value based on the actual processing time; If the predicted fluctuation parameter is greater than the preset predicted fluctuation parameter, the batch selection number of the training data selected for updating the model is adjusted to a corresponding value based on the predicted fluctuation parameter.

4. The dynamic resource optimization method based on large model incremental learning according to claim 3 is characterized in that: The number of servers is adjusted to the corresponding value based on the average evaluation value, where The increase in the number of servers is proportional to the average evaluation value.

5. The dynamic resource optimization method based on large model incremental learning according to claim 4 is characterized in that: The process of determining a data repetition rate based on the data content of the to-be-processed data of a single slice and adjusting the parameters of each server based on the data repetition rate includes: Determine the data sending end corresponding to the data to be processed of a single slice; Get the data sent by the data sender in history; Calculate the repetition rate between each sent data; Solve the average value of each repetition rate to obtain the data repetition rate; If the data repetition rate is less than or equal to the preset data repetition rate, the number of servers is adjusted to a corresponding value based on the average evaluation value; If the data repetition rate is greater than the preset data repetition rate, the single data sending end is marked as an orderly sending end.

6. The dynamic resource optimization method based on large model incremental learning according to claim 5 is characterized in that: Based on the actual processing time, the network bandwidth of the server corresponding to the data to be processed of a single slice is adjusted to the corresponding value, where: The increase in network bandwidth is proportional to the actual processing time.

7. The dynamic resource optimization method based on large model incremental learning according to claim 6 is characterized in that: Based on the predicted fluctuation parameter, the number of batches of training data selected for model update is adjusted to a corresponding value, wherein, The increase in the number of batches selected is proportional to the predicted fluctuation parameter.

8. A dynamic resource optimization system using the dynamic resource optimization method based on large model incremental learning according to any one of claims 1 to 7, characterized in that: include: A data output module, comprising a plurality of data sending terminals for sending data to be processed; a model processing module connected to the data output module, configured to divide the data sent by each data sending end into a plurality of slices of data to be processed, and determine the optimal resource allocation scheme for each server to process the data to be processed and the processing time of each server to process the data to be processed; A data receiving module, which is connected to the data output module and the model processing module respectively, and includes a plurality of servers for receiving and processing the data to be processed of each slice sent by each data sending end according to the optimal resource allocation solution; A data statistics module, connected to the data receiving module, for counting the actual processing time of each server for processing the data to be processed in each slice; A data partitioning module, which is connected to the model processing module, the data receiving module and the data statistics module respectively, and is used to divide the amount of data received by the server, the amount of data processed and the corresponding actual processing time into several batches of training data according to the amount of data; A model training module, which is connected to the data partitioning module and the model processing module respectively, and is used to select several batches of training data to update the model; A data analysis module is respectively connected to the model processing module, the data statistics module, the data receiving module and the model training module, and is used to determine a predicted evaluation value based on the predicted processing time and the actual processing time, judge whether the operating status of the model is qualified based on the predicted evaluation value, and when it is judged that the operating status of the model is abnormal, adjust the parameters of the corresponding server based on the actual processing time of the to-be-processed data of a single slice, including adjusting the number of servers to a corresponding value, adjusting the network bandwidth of the server corresponding to the to-be-processed data of a single slice to a corresponding value, or adjusting the number of batches of training data selected for updating the model to a corresponding value.

Images