A computing platform resource demand prediction method and system
By integrating past computing resource usage logs and timing characteristics at multiple monitoring moments, and using machine learning networks to predict resource demand, the problem of low prediction accuracy in the existing technology is solved, and more accurate and reliable resource demand prediction is achieved.
Patent Information
- Application Number
- CN202411239925.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-05
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-09-05
AI Technical Summary
When facing complex and changing practical application scenarios, the existing computing resource demand forecasting method has low prediction accuracy and fails to fully consider various influencing factors such as business load changes, user behavior patterns and system maintenance activities, resulting in resource Demand forecasts are inaccurate.
By determining the logs of the past computing resource usage at multiple monitoring moments, obtaining the time sequence representation array and the past computing resource representation array, integrating the timing characteristics and resource usage characteristics, using the machine learning network for embedding and restoring mapping, and obtaining the associative representation array. Reason for future resource requirements.
It improves the accuracy and reliability of computing resource demand forecasting, comprehensively considers a variety of influencing factors, and enhances the prediction accuracy.
Smart Images

Figure CN118779113B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a method and system for predicting resource demand for a computing power platform. Background Art
[0002] With the development of cloud computing and big data technology, computing power platforms are increasingly being used in all walks of life. Computing power platforms need to dynamically allocate resources according to different business needs to ensure efficient and stable services. However, due to the volatility and unpredictability of computing power demand, how to accurately predict future computing power resource demand has become an urgent problem to be solved. Traditional computing power resource demand prediction methods are mostly based on single past data for analysis. These methods often ignore the correlation between data at different times, resulting in low accuracy of prediction results. In addition, existing prediction models usually only consider the direct use of computing power resources, but fail to fully consider other influencing factors, such as changes in the external environment and special events, which also limits the accuracy of the prediction. In actual applications, the resource demand of the computing power platform is affected by many factors, including but not limited to changes in business load, changes in user behavior patterns, and the system's own maintenance activities. Therefore, how to comprehensively consider these factors and effectively integrate them into the resource demand prediction model has become the key to improving prediction accuracy. Although the existing computing power resource demand prediction technology can reflect the trend of past data to a certain extent, it often shows inadequacies when facing complex and changeable actual application scenarios. In view of the above problems, there is an urgent need for a new method for predicting computing resource demand that can make full use of the correlation in past data and combine multiple influencing factors to improve the accuracy and reliability of the prediction. Summary of the invention
[0003] In view of this, the present application provides a computing platform resource demand prediction method and system. The technical solution of the present application is implemented as follows:
[0004] On the one hand, the present application provides a computing platform resource demand prediction method, which is applied to a computer system, and the method includes:
[0005] Determine multiple first monitoring moments on the target monitoring data stream, and retrieve the past computing resource usage log corresponding to each first monitoring moment in the multiple first monitoring moments;
[0006] According to the past computing resource usage logs corresponding to the first monitoring moments, obtaining the past computing resource representation arrays corresponding to the first monitoring moments;
[0007] When the timing representation arrays corresponding to the first monitoring moments are obtained, the past computing resource representation arrays and timing representation arrays corresponding to the first monitoring moments are respectively integrated to obtain the first target representation arrays corresponding to the first monitoring moments, wherein the timing representation arrays represent the timing characteristics of the first monitoring moment on the target monitoring data stream;
[0008] According to the first target representation array, obtaining a second target representation array corresponding to the plurality of first monitoring moments, wherein the second target representation array represents the correlation between the past computing resource usage logs corresponding to the respective first monitoring moments;
[0009] Based on the second target representation array, inference is performed on a second monitoring moment on the target monitoring data stream to obtain an inference result of computing power resource usage corresponding to the second monitoring moment, wherein the second monitoring moment is a moment after the first monitoring moment.
[0010] In some implementations, acquiring the second target representation array corresponding to the plurality of first monitoring moments according to the first target representation array includes:
[0011] When one or more linear transformation arrays are obtained, feature mapping is performed on the first target representation arrays corresponding to the respective first monitoring moments through the one or more linear transformation arrays to obtain second target representation arrays corresponding to the multiple first monitoring moments.
[0012] In some implementations, when obtaining one or more linear transformation arrays, feature mapping is performed on the first target representation arrays corresponding to the respective first monitoring moments by using the one or more linear transformation arrays to obtain the second target representation arrays corresponding to the plurality of first monitoring moments, including:
[0013] When obtaining the first linear transformation array and the second linear transformation array, feature mapping is performed on the first target characterization array corresponding to each first monitoring moment through the first linear transformation array and the second linear transformation array, respectively, to obtain the first feature mapping array and the second feature mapping array corresponding to each first monitoring moment, wherein the first feature mapping array is a characterization array obtained by feature mapping the first target characterization array corresponding to each first monitoring moment through the first linear transformation array, and the second feature mapping array is a characterization array obtained by feature mapping the first target characterization array corresponding to each first monitoring moment through the second linear transformation array;
[0014] Respectively performing a combination operation on the first feature mapping arrays corresponding to the first monitoring moments and the second feature mapping arrays corresponding to the first monitoring moments to obtain a first linear transformation array corresponding to the first feature mapping array and a second linear transformation array corresponding to the second feature mapping array;
[0015] The first linear transformation array and the second linear transformation array are processed simultaneously to obtain a second target representation array corresponding to the multiple first monitoring moments.
[0016] In some implementations, obtaining the past computing resource representation array corresponding to each first monitoring moment according to the past computing resource usage log corresponding to each first monitoring moment includes:
[0017] Obtaining the impact factor data corresponding to the past computing resource usage logs corresponding to the first monitoring moments, wherein the impact factor data is dynamically changing control data associated with the past computing resource usage logs;
[0018] Performing a combination operation on the past computing resource usage logs corresponding to each of the first monitoring moments and the corresponding influencing factor data, to obtain a past computing resource representation array corresponding to each of the first monitoring moments;
[0019] Before respectively fusing the past computing resource representation arrays and the timing representation arrays corresponding to the first monitoring moments to obtain the first target representation arrays corresponding to the first monitoring moments, the method further includes:
[0020] The timing characteristics of each first monitoring moment are embedded through wave functions of different wavelengths to obtain a timing representation array corresponding to each first monitoring moment.
[0021] In some embodiments, after determining a plurality of first monitoring moments on the target monitoring data stream, and retrieving a past computing resource usage log corresponding to each of the plurality of first monitoring moments, the method further includes:
[0022] Loading the past computing power resource usage logs corresponding to each of the first monitoring moments into a target machine learning network, wherein the target machine learning network is a neural network obtained by debugging a learning example, and the target machine learning network is used to complete computing power resource demand reasoning, and the target machine learning network includes an embedding mapping component and a restoring mapping component;
[0023] Embedding the past computing resource usage logs corresponding to each of the first monitoring moments by the embedding mapping component to obtain the second target representation array;
[0024] The second target representation array is restored and mapped by the restoration mapping component to obtain the computing power resource usage inference result.
[0025] In some implementations, performing restoration mapping on the second target representation array by the restoration mapping component to obtain the computing power resource usage inference result includes:
[0026] The second target representation array is mapped into a mapping result of a preset dimension through a dense network component in the restoration mapping component, and the mapping result is used as the computing power resource usage inference result, wherein the dense network component is used to map the embedded information output by the embedding mapping component into data of the preset dimension.
[0027] In some implementations, before loading the past computing resource usage logs corresponding to each first monitoring moment into the target machine learning network, the method further includes:
[0028] Acquire multiple debugging learning samples and determine a target machine learning network to be debugged, wherein the debugging learning samples include a sample past computing resource usage log at a first sample monitoring time and a test past computing resource usage log at a second sample monitoring time, wherein the second sample monitoring time is a time after the first sample monitoring time;
[0029] Loading the multiple debugging learning samples into the target machine learning network to be debugged, and obtaining multiple mapping results of the target machine learning network to be debugged, wherein the mapping results are data inferred by the target machine learning network to be debugged based on the past computing resource usage logs of the samples;
[0030] Determine a test past computing resource usage log corresponding to each mapping result in the multiple mapping results, and determine the current loss of the target machine learning network to be debugged based on the mapping result and the corresponding test past computing resource usage log;
[0031] If the current loss meets the preset judgment criteria, the target machine learning network to be debugged is determined as the target machine learning network;
[0032] If the current loss does not meet the preset judgment criteria, the network parameters of the target machine learning network to be debugged are optimized, and the optimized target machine learning network is determined as the target machine learning network to be debugged.
[0033] In some implementations, determining the current loss of the target machine learning network to be debugged based on the mapping result and the corresponding test past computing resource usage log includes:
[0034] Based on the preset multiple loss functions, the loss of the mapping result and the corresponding test past computing resource usage log is jointly determined to obtain the current loss.
[0035] In some implementations, before determining a plurality of first monitoring moments on the target monitoring data stream and retrieving a past computing resource usage log corresponding to each of the plurality of first monitoring moments, the method further includes:
[0036] Obtaining a computing power demand estimation command generated for a preset computing power platform, wherein the computing power demand estimation command is used to trigger inference of the computing power demand of the preset computing power platform at the second monitoring time, and the target monitoring data stream is the monitoring data stream of the preset computing power platform;
[0037] The determining of a plurality of first monitoring moments on the target monitoring data stream, and retrieving a past computing resource usage log corresponding to each of the plurality of first monitoring moments, includes:
[0038] Based on the computing power demand estimation command, the multiple first monitoring moments associated with the second monitoring moment on the target monitoring data stream are determined, and the past computing power resource usage logs corresponding to the respective first monitoring moments are retrieved.
[0039] In a second aspect, the present application provides a computer system, comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor implements the steps in the above-described method when executing the program.
[0040] The technical effects of the present application include: in the embodiment of the present application, multiple first monitoring moments on the target monitoring data stream are determined, and the past computing resource usage logs corresponding to each of the above multiple first monitoring moments are retrieved, and the past computing resource representation arrays corresponding to each of the above multiple first monitoring moments are obtained based on the past computing resource usage logs corresponding to each of the above multiple first monitoring moments. When obtaining the time series representation arrays corresponding to each of the above multiple first monitoring moments, the past computing resource representation arrays and the time series representation arrays corresponding to each of the above multiple first monitoring moments are respectively integrated to obtain the first target representation arrays corresponding to each of the above multiple first monitoring moments, wherein the time series representation array represents the time series characteristics of the above multiple first monitoring moments on the above multiple first monitoring moments, and the second target representation arrays corresponding to the above multiple first monitoring moments are obtained based on the above multiple first target representation arrays. The above multiple first target representation arrays represent the correlation between the past computing resource usage logs corresponding to each of the above multiple first monitoring moments, and the above multiple first monitoring moments are inferred based on the above multiple first target representation arrays. The above multiple first monitoring moments are inferred based on the above multiple first monitoring moments, and the computing resource usage inference result corresponding to the above multiple first monitoring moments is obtained, and the above multiple first monitoring moments are inferred based on the above multiple first target representation arrays. The above multiple first monitoring moments are inferred based on the above multiple first monitoring moments ...
[0041] Through the above computing power platform resource demand prediction method, multiple past monitoring moments are determined according to the target monitoring data stream, and the past computing power resource representation array corresponding to each past monitoring moment and the time series representation array corresponding to each past monitoring moment are obtained and respectively integrated to obtain the first target representation array corresponding to each past monitoring moment. Based on the first target representation array, a second target representation array representing the correlation between the past computing power resource usage logs corresponding to each past monitoring moment is obtained. Based on the correlation between the past computing power resource usage logs, the defect of insufficient data richness of the past computing power resource usage logs used for computing power resource usage reasoning results is overcome, and the demand prediction accuracy of the second target representation array is increased. The second target representation array is used to infer the computing power resource usage reasoning results corresponding to the target monitoring moments after multiple past monitoring moments on the target monitoring data stream, so as to achieve the correlation of the past computing power resource usage logs corresponding to the multiple past monitoring moments on the target monitoring data stream, increase the accuracy of the computing power resource usage reasoning results corresponding to the target monitoring moments after multiple past monitoring moments, and increase the accuracy of the obtained computing power resource usage reasoning results.
[0042] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.
[0044] Figure 1 A schematic diagram of the implementation flow of a computing power platform resource demand prediction method provided in an embodiment of the present application.
[0045] Figure 2 A hardware entity diagram of a computer system provided in an embodiment of the present application. DETAILED DESCRIPTION
[0046] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application are further elaborated in detail below in conjunction with the drawings and embodiments. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.
[0047] The embodiment of the present application provides a method for predicting resource demand of a computing platform, which can be executed by a processor of a computer system, wherein the computer system can refer to a cloud computing server or other data processing devices in the computing platform.
[0048] Figure 1 A schematic diagram of the implementation flow of a computing power platform resource demand prediction method provided in an embodiment of the present application is as follows: Figure 1 As shown, the method comprises the following steps:
[0049] Step S100: determine a plurality of first monitoring moments on a target monitoring data stream, and retrieve a past computing resource usage log corresponding to each of the plurality of first monitoring moments.
[0050] When the computer system executes step S100, it first determines multiple first monitoring moments on the target monitoring data stream. For example, the computer system selects the peak hours of each day in the past week as the first monitoring moment, and these peak hours are usually between 12:00 and 13:00 every day. Next, the system retrieves the past computing power resource usage logs corresponding to each of these first monitoring moments. These logs record the past computing power resource usage, including but not limited to key performance indicators such as CPU utilization, memory utilization, storage space occupancy, and network bandwidth usage.
[0051] To explain more specifically, assume that the computer system extracts the computing resource usage logs from the past database between 12:00 and 13:00 every day from August 1 to August 7, 2024. Each log is a representation array. For example, for the log on August 1, 2024, the representation array may be [80, 65, 70, 30], where 80 represents 80% CPU utilization, 65 represents 65% memory utilization, 70 represents 70% storage space utilization, and 30 represents 30% network bandwidth utilization.
[0052] In addition, the computer system needs to call the past computing resource usage logs corresponding to the first monitoring moment and load them into the target machine learning network. The target machine learning network is a trained neural network model, and its structure may include an input layer, a hidden layer, and an output layer. The input layer receives the feature vectors in the computing resource usage logs, the hidden layer is responsible for performing complex nonlinear transformations on the input data, and the output layer generates the computing resource usage inference results.
[0053] During the training phase, the computer system uses multiple debugging learning examples to adjust the weights and bias parameters of the target machine learning network to minimize the loss function. The loss function can be a mean squared error formula, such as , where L is the loss value and y i is the actual usage of computing resources. is the network prediction value, and N is the number of samples. Through continuous iterative optimization, until the value of the loss function drops below the predetermined threshold, the target machine learning network is considered to be successfully trained.
[0054] Step S200: Obtaining a past computing resource representation array corresponding to each first monitoring moment according to the past computing resource usage log corresponding to each first monitoring moment.
[0055] When the computer system executes step S200, it obtains the past computing resource representation array corresponding to each first monitoring moment according to the past computing resource usage log corresponding to each first monitoring moment. This process involves extracting useful information from the past computing resource usage log to form a representation array for subsequent analysis.
[0056] Specifically, assume that the computer system has determined multiple first monitoring moments on the target monitoring data stream. For example, 12:00 to 13:00 every day from August 1 to August 7, 2024 is selected as the first monitoring moment. For each selected first monitoring moment, the system will retrieve the corresponding computing resource usage log. These logs record in detail the key performance indicators such as CPU utilization, memory utilization, storage space occupancy, network bandwidth usage, etc. at each moment.
[0057] For further explanation, taking August 1, 2024 from 12:00 to 13:00 as an example, the data recorded in the corresponding computing resource usage log may be: CPU utilization is 80%, memory utilization is 65%, storage space utilization is 70%, and network bandwidth utilization is 30%. The computer system converts this data into a representation array, such as [80, 65, 70, 30].
[0058] In addition, the system needs to obtain the impact factor data corresponding to each first monitoring moment, which is associated with the past computing resource usage log and is a dynamically changing control data. For example, the impact factor data may include the weather conditions of the day, special event arrangements, etc., which may indirectly affect the demand for computing resources. Assuming that the weather is sunny on August 1 and there are no special events scheduled, the impact factor data may be represented as [0, 0], where 0 indicates no special impact. Next, the computer system combines the past computing resource usage logs corresponding to each first monitoring moment and the corresponding impact factor data to obtain the past computing resource representation array corresponding to each first monitoring moment. Continuing with the above example, for August 1 from 12:00 to 13:00, the final representation array will be [80, 65, 70, 30, 0, 0]. In this way, the computer system can comprehensively capture the specific computing resource usage status and its potential influencing factors at each monitoring moment, providing rich data support for subsequent steps. These representation arrays will be used for further analysis to predict the computing resource demand at future moments.
[0059] Step S300: When the timing representation array corresponding to each first monitoring moment is obtained, the past computing resource representation array and the timing representation array corresponding to each first monitoring moment are respectively integrated to obtain the first target representation array corresponding to each first monitoring moment, and the timing representation array represents the timing characteristics of the first monitoring moment on the target monitoring data stream.
[0060] When the computer system executes step S300, after obtaining the timing representation array corresponding to each first monitoring moment, the computer system respectively fuses the past computing resource representation array corresponding to each first monitoring moment with the timing representation array to obtain the first target representation array corresponding to each first monitoring moment. The timing representation array reflects the timing characteristics of the first monitoring moment on the target monitoring data stream, that is, the position characteristics.
[0061] For example, assume that the computer system has determined multiple first monitoring moments on the target monitoring data stream. For example, 12:00 to 13:00 every day from August 1 to August 7, 2024 is selected as the first monitoring moment. For each first monitoring moment, the computer system has obtained the corresponding past computing resource representation array. For example, the representation array on August 1 is [80, 65,70, 30], indicating that the CPU utilization rate is 80%, the memory utilization rate is 65%, the storage space occupancy rate is 70%, and the network bandwidth utilization rate is 30%. In order to embed the timing features, the computer system embeds the timing features of each first monitoring moment. For example, different wave functions are used to represent the position features of different monitoring moments. For the moment of 12:00 on August 1, assuming that the sine wave function and the cosine wave function are used to represent the time and date respectively, then its timing representation array may be [sin(1), cos(1)], where the number 1 represents the position code of the moment in the monitoring data stream. The computer system fuses the above computing power resource representation array with the time series representation array to form a first target representation array. Continuing with August 1st at 12:00 as an example, the fused first target representation array will be [80, 65, 70, 30, sin(1), cos(1)]. Such a fusion operation ensures that the representation array not only contains the specific computing power resource usage, but also adds the position information of the moment in the entire data stream, thereby better reflecting the pattern in the time series. In this way, the computer system can more comprehensively capture the characteristics of each monitoring moment and provide a more accurate data basis for subsequent computing power resource demand prediction. The first target representation array will be used in the next step to obtain the second target representation array corresponding to each first monitoring moment, and then used to predict the computing power resource demand at future moments.
[0062] Step S400: According to the first target characterization array, obtain a second target characterization array corresponding to the multiple first monitoring moments, wherein the second target characterization array represents the correlation between the past computing power resource usage logs corresponding to the respective first monitoring moments.
[0063] When the computer system executes step S400, it obtains the second target characterization array corresponding to multiple first monitoring moments according to the first target characterization array. The second target characterization array characterizes the correlation between the past computing power resource usage logs corresponding to each first monitoring moment. Specifically, the computer system first obtains the first target characterization array corresponding to each first monitoring moment. Assume that the system selects 12:00 to 13:00 every day from August 1 to August 7, 2024 as the first monitoring moment, and has obtained the first target characterization array formed by the fusion of the computing power resource characterization array and the time series characterization array of each day. For example, the first target characterization array on August 1 is [80, 65, 70, 30, sin(1), cos(1)], where the first four elements represent the computing power resource usage and the last two elements represent the location characteristics of the moment. In order to obtain the second target characterization array, the computer system needs to perform feature mapping on the first target characterization array. This process can be achieved by one or more linear transformation arrays. For example, two parameter matrices W1 and W2 are used to perform feature mapping on the first target characterization array. For the first target representation array [80, 65, 70, 30, sin(1), cos(1)] on August 1, the first feature mapping array X1 is obtained by mapping W1, and the second feature mapping array X2 is obtained by mapping W2. Assuming that W1 is a 2x6 matrix and W2 is also a 2×6 matrix, the mapping process can be expressed as:
[0064] ;
[0065] ;
[0066] Next, the computer system performs a combination operation on the first feature mapping array and the second feature mapping array corresponding to each first monitoring moment to obtain the first linear transformation array and the second linear transformation array corresponding to each first monitoring moment. For example, for August 1, the result after combination is [X1, X2].
[0067] Then, the computer system performs simultaneous processing based on the first linear transformation array and the second linear transformation array to obtain a second target representation array corresponding to multiple first monitoring moments. For example, for all seven monitoring moments, the system combines the respective first linear transformation arrays and second linear transformation arrays to form a new representation array, namely the second target representation array. The second target representation array reflects the correlation between the first monitoring moments. This correlation helps the system understand the mutual influence between different monitoring moments, so as to better predict the computing power resource requirements at future moments. For example, if the computing power resource usage shows a similar trend in the same time period for several consecutive days, then this trend will be reflected in the second target representation array, helping the computer system to more accurately predict resource requirements.
[0068] In this way, the computer system can comprehensively consider the past data of multiple first monitoring moments and explore the intrinsic connections between these data, thereby improving the accuracy and reliability of computing resource demand prediction.
[0069] Step S500: Based on the second target characterization array, infer the second monitoring time on the target monitoring data stream to obtain the computing power resource usage inference result corresponding to the second monitoring time, wherein the second monitoring time is a time after the first monitoring time.
[0070] When the computer system executes step S500, it infers the second monitoring moment on the target monitoring data stream based on the second target characterization array, and obtains the inference result of the computing power resource usage corresponding to the second monitoring moment. The second monitoring moment refers to the moment after the first monitoring moment. Specifically, the computer system has obtained the second target characterization arrays corresponding to each first monitoring moment, and these characterization arrays reflect the correlation between the computing power resource usage between different monitoring moments. For example, for the first monitoring moment from 12:00 to 13:00 every day from August 1 to August 7, 2024, the system has obtained the second target characterization array corresponding to each moment. Assume that the second target characterization array on August 1 is [80.5, 66.2,71.3, 30.4, 0.5, -0.2], where the first four elements represent the mapping results of the computing power resource usage, and the last two elements represent the mapping results of the time series features. In order to infer the second monitoring moment on the target monitoring data stream, the computer system uses the target machine learning network to process these second target characterization arrays. The target machine learning network is a neural network model trained by debugging learning examples. For example, the model may include an embedding mapping component and a restoration mapping component. The embedding mapping component is responsible for mapping the second target representation array to a higher dimensional space to capture more complex feature relationships; the restoration mapping component maps the features in the high-dimensional space back to the original computing resource usage. Continuing with August 1 as an example, the computer system inputs the second target representation array [80.5, 66.2, 71.3, 30.4, 0.5, -0.2] on August 1 into the embedding mapping component of the target machine learning network. The embedding mapping component maps the input array to a higher dimensional feature space through a series of nonlinear transformations. For example, the embedding mapping component may contain a multi-layer fully connected neural network, each layer with several neurons. Assume that after the embedding mapping component, the embedded information obtained is a vector E1 of length 10.
[0071] Next, the computer system passes E1 to the reduction mapping component. The reduction mapping component is responsible for mapping the embedded information back to the original computing resource usage. For example, the reduction mapping component may be a dense network component that maps E1 into data of a preset dimension. Assuming that after the reduction mapping component, the computing resource usage inference result obtained is [82, 67,72, 31], it means that the computer system predicts the computing resource usage from 12:00 to 13:00 on August 8 as follows: CPU utilization is 82%, memory utilization is 67%, storage space utilization is 72%, and network bandwidth utilization is 31%. In this way, the computer system can predict the computing resource demand at future times based on past data and current computing resource usage. This prediction not only takes into account past resource usage, but also combines time series characteristics and other possible influencing factors to improve the accuracy and reliability of the prediction.
[0072] As an implementation manner, the step S400, obtaining the second target representation array corresponding to the plurality of first monitoring moments according to the first target representation array, includes:
[0073] Step S410: When one or more linear transformation arrays are obtained, feature mapping is performed on the first target representation arrays corresponding to the first monitoring moments through the one or more linear transformation arrays to obtain the second target representation arrays corresponding to the multiple first monitoring moments.
[0074] When the computer system executes step S410, when obtaining one or more linear transformation arrays, the first target characterization arrays corresponding to each first monitoring moment are feature mapped through these linear transformation arrays to obtain the second target characterization arrays corresponding to multiple first monitoring moments. Specifically, the computer system first has the first target characterization arrays corresponding to each first monitoring moment. Assume that 12:00 to 13:00 every day from August 1 to August 7, 2024 has been determined as the first monitoring moment, and the first target characterization array of each day has been obtained. For example, the first target characterization array on August 1 is [80, 65, 70, 30, sin(1), cos(1)], where the first four elements represent the usage of computing resources and the last two elements represent the location characteristics of the moment. In order to obtain the second target characterization array, the computer system needs to use one or more linear transformation arrays (such as parameter matrices) to perform feature mapping on the first target characterization array. For example, two parameter matrices W1 and W2 are used to map the first target characterization array. Assuming W1 is a 2×6 matrix and W2 is also a 2x6 matrix, the mapping process can be expressed as:
[0075] ;
[0076] ;
[0077] Here, W1 and W2 are parameter matrices, and X1 and X2 are the first feature map array and the second feature map array, respectively. For example, W1 may be:
[0078] ;
[0079] Among them, W 11 ~W 26 is a parameter element in the parameter matrix; through matrix multiplication, W1 is multiplied by the first target representation array to obtain X1. Similarly, W2 is multiplied by the first target representation array to obtain X2.
[0080] Next, the computer system performs a combination operation on the first feature mapping array X1 and the second feature mapping array X2 corresponding to each first monitoring moment. For example, for August 1, the result of the combination is [X1, X2]. Then, the computer system performs simultaneous processing based on the first linear transformation array X1 and the second linear transformation array X2 to obtain a second target representation array corresponding to multiple first monitoring moments. For example, for all seven monitoring moments, the system combines the respective first linear transformation arrays and the second linear transformation arrays to form a new representation array, namely the second target representation array.
[0081] The second target characterization array reflects the correlation between each first monitoring moment. This correlation helps the system understand the mutual influence between different monitoring moments, so as to better predict the computing power resource demand at future moments. For example, if the computing power resource usage shows a similar trend in the same time period for several consecutive days, then this trend will be reflected in the second target characterization array, helping the computer system to predict resource demand more accurately. In this way, the computer system can comprehensively consider the past data of multiple first monitoring moments and dig out the intrinsic connection between these data, thereby improving the accuracy and reliability of the computing power resource demand prediction.
[0082] As an implementation manner, in step S410, when obtaining one or more linear transformation arrays, feature mapping is performed on the first target representation arrays corresponding to the first monitoring moments by using the one or more linear transformation arrays to obtain the second target representation arrays corresponding to the multiple first monitoring moments, which may include:
[0083] Step S411: when obtaining the first linear transformation array and the second linear transformation array, feature mapping is performed on the first target representation array corresponding to each first monitoring moment through the first linear transformation array and the second linear transformation array respectively, so as to obtain the first feature mapping array and the second feature mapping array corresponding to each first monitoring moment, wherein the first feature mapping array is a representation array obtained by feature mapping the first target representation array corresponding to each first monitoring moment through the first linear transformation array, and the second feature mapping array is a representation array obtained by feature mapping the first target representation array corresponding to each first monitoring moment through the second linear transformation array;
[0084] Step S412: performing a combination operation on the first feature mapping arrays corresponding to the first monitoring moments and the second feature mapping arrays corresponding to the first monitoring moments, respectively, to obtain a first linear transformation array corresponding to the first feature mapping array and a second linear transformation array corresponding to the second feature mapping array;
[0085] Step S413: performing simultaneous processing according to the first linear transformation array and the second linear transformation array to obtain a second target representation array corresponding to the plurality of first monitoring moments.
[0086] When the computer system executes step S410, the first target characterization array corresponding to each first monitoring moment is feature mapped by the obtained first linear transformation array and second linear transformation array to obtain second target characterization arrays corresponding to multiple first monitoring moments. This process is divided into three sub-steps: S411, S412 and S413.
[0087] In step S411, the computer system first obtains a first linear transformation array and a second linear transformation array. These linear transformation arrays are usually in the form of parameter matrices, which are used to perform feature mapping on the first target characterization array. Assume that the computer system has determined 12:00 to 13:00 every day from August 1 to August 7, 2024 as the first monitoring moment, and has obtained the first target characterization array for each day. For example, the first target characterization array on August 1 is [80, 65, 70,30, sin(1), cos(1)], where the first four elements represent the usage of computing resources, and the last two elements represent the location characteristics at that moment.
[0088] In order to perform feature mapping, the computer system uses two parameter matrices W1 and W2. Assume that W1 is a 2x6 matrix and W2 is also a 2x6 matrix. Through these two parameter matrices, the computer system performs feature mapping on the first target characterization array at each first monitoring moment. For example, for the first target characterization array on August 1, the mapping process can be expressed as:
[0089] ;
[0090] ;
[0091] Here, W1 and W2 are parameter matrices, X1 and X2 are the first feature map array and the second feature map array, respectively. Assume that the mapped results are X1=[0.8, 0.6], X2=[0.7, 0.3].
[0092] In step S412, the computer system performs a combination operation on the first feature mapping array and the second feature mapping array corresponding to each first monitoring moment. Specifically, for the first feature mapping array and the second feature mapping array of each day from August 1 to August 7, the system combines them into new arrays. For example, assuming that the first feature mapping arrays from August 1 to August 7 are [0.8, 0.6], [0.75, 0.65], [0.7, 0.6], [0.75, 0.65], [0.8, 0.7], [0.85, 0.75], [0.9, 0.8], and the second feature mapping arrays are [0.7, 0.3], [0.65, 0.35], [0.6,0.4], [0.65, 0.35], [0.7, 0.4], [0.75, 0.45], [0.8, 0.5]. The system combines these arrays into two new arrays, namely:
[0093] ;
[0094] ;
[0095] In step S413, the computer system transforms the first linear array and the second linear transformation array The computer system can process the first monitoring moments simultaneously to obtain a second target representation array corresponding to the first monitoring moments. This processing may involve further linear combination or other forms of feature fusion. For example, the computer system can and Merge into a larger vector or use another neural network layer to further process the features.
[0096] Assume that the computer system uses a simple linear combination method to and Merge into the second target representation array Y:
[0097] ;
[0098] In this way, the second target representation array Y finally formed will contain the feature mapping information of all the first monitoring moments, reflecting the correlation between these moments.
[0099] In this way, the computer system can comprehensively consider the past data of multiple first monitoring moments and dig out the internal connections between these data, thereby improving the accuracy and reliability of computing resource demand prediction. The second target representation array Y will be used for subsequent computing resource demand reasoning to predict computing resource demand at future moments.
[0100] As an implementation manner, the step S200, obtaining the past computing resource representation array corresponding to each first monitoring moment according to the past computing resource usage log corresponding to each first monitoring moment, may include:
[0101] Step S210: Obtaining the impact factor data corresponding to the past computing resource usage logs corresponding to the first monitoring moments, wherein the impact factor data is dynamically changing control data associated with the past computing resource usage logs;
[0102] Step S220: performing a combination operation on the past computing power resource usage logs corresponding to the respective first monitoring moments and the respective corresponding influencing factor data to obtain a past computing power resource representation array corresponding to the respective first monitoring moments.
[0103] When the computer system executes step S200, based on the past computing resource usage log corresponding to each first monitoring moment, it obtains the past computing resource representation array corresponding to each first monitoring moment. This process is divided into two sub-steps: S210 and S220.
[0104] In step S210, the computer system obtains the impact factor data corresponding to the past computing power resource usage logs corresponding to each first monitoring moment. These impact factor data are dynamically changing control data associated with the past computing power resource usage logs. For example, assume that the computer system has determined 12:00 to 13:00 every day from August 1 to August 7, 2024 as the first monitoring moment, and has obtained the computing power resource usage logs for each day. These logs record in detail the key performance indicators such as CPU utilization, memory utilization, storage space occupancy, network bandwidth usage, etc. at each moment.
[0105] To explain more specifically, assume that the computer system extracts the computing resource usage logs from the past database between 12:00 and 13:00 every day from August 1 to August 7, 2024. Each log is a representation array. For example, for the log on August 1, 2024, the representation array may be [80, 65, 70, 30], where 80 represents 80% CPU utilization, 65 represents 65% memory utilization, 70 represents 70% storage space utilization, and 30 represents 30% network bandwidth utilization.
[0106] Next, the computer system obtains the impact factor data corresponding to each first monitoring moment. These data are associated with the past computing resource usage logs and are dynamically changing control data. For example, the impact factor data may include the weather conditions of the day, special event arrangements, etc. These factors may indirectly affect the demand for computing resources. Assuming that the weather is sunny on August 1 and there are no special events scheduled, the impact factor data may be expressed as [0, 0], where 0 means no special impact.
[0107] In step S220, the computer system combines the past computing resource usage logs and the corresponding influencing factor data corresponding to each first monitoring moment to obtain the past computing resource representation array corresponding to each first monitoring moment. Specifically, the computer system combines the computing resource usage logs and the corresponding influencing factor data at each first monitoring moment into a new representation array. Continuing with August 1 as an example, for August 1 from 12:00 to 13:00, the final representation array will be [80, 65, 70, 30, 0, 0].
[0108] Assume that the situation on August 2 is slightly different. The computing resource usage log for that day is [75, 60, 75, 25], and there is a large online event on that day, which leads to additional computing power requirements. Therefore, the impact factor data is [1, 0], indicating that there is a special event but the weather is still sunny. Then, the final representation array from 12:00 to 13:00 on August 2 is [75, 60, 75, 25,1, 0].
[0109] In this way, the computer system can comprehensively capture the specific computing resource usage and its potential influencing factors at each monitoring moment, providing rich data support for subsequent steps. These representation arrays will be used for further analysis to predict computing resource requirements at future moments.
[0110] To further illustrate the whole process, assume that the computer system needs to process data within a week, as follows:
[0111] The representation array for August 1st from 12:00 to 13:00 is [80, 65, 70, 30, 0, 0].
[0112] The representation array for August 2 from 12:00 to 13:00 is [75, 60, 75, 25, 1, 0].
[0113] The representation array for August 3rd from 12:00 to 13:00 is [85, 70, 75, 35, 0, 0].
[0114] The representation array from 12:00 to 13:00 on August 4 is [82, 68, 72, 32, 0, 0].
[0115] The representation array from 12:00 to 13:00 on August 5 is [80, 65, 70, 30, 0, 0].
[0116] The representation array from 12:00 to 13:00 on August 6 is [78, 63, 71, 28, 0, 0].
[0117] The representation array from 12:00 to 13:00 on August 7 is [81, 66, 73, 31, 0, 0].
[0118] In this way, the computer system can generate a comprehensive representation array for each monitoring moment, which includes the computing resource usage and other factors that may affect the computing power demand. These representation arrays will be used in subsequent steps to further integrate time series features and make computing power resource demand predictions.
[0119] For example, in the subsequent steps, the computer system will use these representation arrays to obtain the time series representation arrays corresponding to each first monitoring moment, and then merge them into the first target representation array, which is finally used to predict the demand for computing power resources. This method of comprehensively considering multiple factors can improve the accuracy and reliability of the prediction and ensure that the computing power platform can maintain normal operation in the future.
[0120] As an implementation manner, in step S300, before respectively fusing the past computing resource representation array and the time series representation array corresponding to each first monitoring moment to obtain the first target representation array corresponding to each first monitoring moment, the method may further include:
[0121] The timing characteristics of each first monitoring moment are embedded through wave functions of different wavelengths to obtain a timing representation array corresponding to each first monitoring moment.
[0122] When the computer system executes step S300, it will first embed the time series characteristics of each first monitoring moment through the wave function of different wavelengths, so as to obtain the time series representation array corresponding to each first monitoring moment. This process is to enhance the time dimension information of the time series data before integrating the past computing power resource representation array, so that the model can better capture the law of computing power resource demand changing over time.
[0123] The computer system first defines a series of wave functions with different wavelengths, which are used to embed time series features. For example, the computer system can select the sine wave function sin(ωt +φ), where ω represents the angular frequency, t is the time variable, and φ is the phase shift. Different ω values represent different cycle lengths, i.e., wavelengths. By selecting different ω values, the computer system can capture seasonal changes of different cycles, such as cycle changes within a day, cycle changes per week, and cycle changes per month.
[0124] Next, the computer system applies the above wave function to the data at each first monitoring moment. Assume that the computer system has obtained the computing resource usage log and its influencing factor data from 12:00 to 13:00 every day from August 1 to August 7, 2024, and combined them into a characterization array. To simplify the explanation, only examples of August 1 and August 2 are shown here. The characterization array from 12:00 to 13:00 on August 1 is [80, 65, 70, 30, 0, 0], and the characterization array from 12:00 to 13:00 on August 2 is [75, 60, 75, 25, 1, 0].
[0125] The computer system then applies multiple wave functions with different ω values to process these representation arrays. For example, for periodic changes within a day, ω1=2π / 24h can be selected; for periodic changes in a week, ω2=2π / 168h is selected (a week has 168 hours). After applying these wave functions, the original representation array is mapped to a higher-dimensional space to form a time series representation array. This process can be understood as a feature transformation of the original representation array, adding information in the time dimension.
[0126] Specifically, the computer system can calculate the new time series features by the following formula: F'(t) = [F(t), sin(ω1t + φ1), cos(ω1t + φ1), sin(ω2t + φ2), cos(ω2t + φ2)]. Where F(t) is the original representation array, F'(t) is the time series representation array, and the sin and cos terms represent the periodic features of different periods. Through such a transformation, the original representation array [80, 65, 70, 30, 0, 0] may be expanded to a longer array that includes the original features and new features generated by different wave functions.
[0127] After the embedding of the time series features is completed, the computer system obtains the time series representation arrays corresponding to each first monitoring moment. These arrays will be merged with the past computing power resource representation arrays to form the first target representation array for subsequent computing power resource demand prediction.
[0128] For example, for August 1st from 12:00 to 13:00, the timing representation array of the computer system after wave function embedding may be [80,65,70,30,0,0,sin(ω1*t+φ1),cos(ω1*t+φ1),sin(ω2*t+φ2),cos(ω2*t+φ2)], where the values of sin and cos depend on the specific time t. For August 2nd from 12:00 to 13:00, the corresponding timing representation array is [75,60,75,25,1,0,sin(ω1*t+φ1),cos(ω1*t+φ1),sin(ω2*t+φ2),cos(ω2*t+φ2)].
[0129] Finally, the computer system merges these time series representation arrays with the previously obtained past computing resource representation arrays to form the first target representation array, which will be used to train the prediction model to make accurate predictions of computing resource requirements in the future. In this way, the computer system can make full use of time series features and improve the performance of the prediction model.
[0130] As an implementation mode, in the step S100, that is, after determining a plurality of first monitoring moments on the target monitoring data stream, and retrieving a past computing resource usage log corresponding to each of the plurality of first monitoring moments, the method may further include:
[0131] Step S101: Loading the past computing power resource usage logs corresponding to the first monitoring moments into a target machine learning network, wherein the target machine learning network is a neural network obtained by debugging a learning example, and the target machine learning network is used to complete computing power resource demand reasoning, and the target machine learning network includes an embedding mapping component and a restoring mapping component;
[0132] Step S102: embedding the past computing resource usage logs corresponding to each of the first monitoring moments through the embedding mapping component to obtain the second target representation array;
[0133] Step S103: performing restoration mapping on the second target representation array through the restoration mapping component to obtain the computing power resource usage inference result.
[0134] In the process of executing step S100, the computer system determines multiple first monitoring moments on the target monitoring data stream through a series of sub-steps, and further retrieves the past computing resource usage logs corresponding to each first monitoring moment. On this basis, the computer system continues to perform operations from steps S101 to S103, that is, the collected log data is processed through the pre-debugged target machine learning network to infer future computing resource requirements.
[0135] In step S101, the computer system loads the past computing power resource usage logs corresponding to each first monitoring moment into the target machine learning network. The target machine learning network is a neural network model trained by debugging learning samples, which is specifically used to complete the computing power resource demand reasoning task. Specifically, the network includes two key parts: the embedding mapping component and the restoration mapping component. The embedding mapping component is responsible for converting the original data into a form suitable for model processing, while the restoration mapping component is responsible for converting the intermediate representation into the final computing power resource usage reasoning result.
[0136] Step S102 involves embedding the past computing resource usage logs corresponding to each first monitoring moment through an embedding mapping component to obtain a second target representation array. Taking a specific example to illustrate this process, assuming that a first monitoring moment is from 12:00 to 13:00 on August 10, 2024, the corresponding past computing resource usage log contains the following types of information: CPU utilization, memory utilization, network traffic, storage usage, number of tasks running, and computing resource demand in the same time period in the past. These data can be organized into a multidimensional array form, such as [80, 75, 1500, 70, 20, 90], where each element corresponds to a specific value of the above six types of information. The computer system converts this original multidimensional array into a vector in a higher dimensional vector space, i.e., the second target representation array, through an embedding mapping component. This conversion process usually involves preprocessing steps such as normalization, noise reduction, feature extraction, and possible nonlinear transformation of the original data to make the data more suitable for neural network learning. For example, by using sequence models such as Long Short-Term Memory (LSTM) or Gated Recurrent Unit (GRU) as part of the embedding mapping component, long-term dependencies in time series data can be captured.
[0137] Entering step S103, the computer system processes the second target representation array obtained in the previous step through the restoration mapping component, and finally obtains the computing power resource usage inference result. The working principle of the restoration mapping component is that it decodes the highly abstract representation array obtained from the embedding mapping component back to a human-interpretable form, usually predicting the computing power resource demand value within a specific time period in the future. This step may involve the use of activation functions, regression analysis and other techniques to output the final prediction value. In this example, it is assumed that after the above steps, the computer system predicts that the computing power resource demand from 12:00 to 13:00 on August 17, 2024 is [85, 70, 1600, 75, 25, 95], which indicates that compared with a week ago, except for a slight decrease in memory usage, all other indicators have increased.
[0138] In the whole process, the choice of loss function is crucial in the training of the neural network model used by the computer system, which determines the direction of model optimization. For example, mean square error (MSE) or mean absolute error (MAE) is often used as the loss function in regression problems. Taking mean square error as an example, its calculation formula is , where L is the loss value, y i Indicates the actual computing resource demand value, Represents the computing power resource demand value predicted by the model, and n represents the number of samples. By minimizing this loss function, the computer system can adjust the model parameters so that the predicted value is as close to the true value as possible, thereby improving the accuracy of the computing power resource demand prediction. In summary, through the operations of steps S101 to S103, the computer system effectively completes the conversion from past computing power resource usage logs to future computing power resource demand predictions using the pre-trained target machine learning network, providing an important decision-making basis for the computing power platform.
[0139] As an implementation mode, the step S103, i.e., performing restoration mapping on the second target representation array through the restoration mapping component to obtain the computing power resource usage inference result, may include: mapping the second target representation array into a mapping result of a preset dimension through a dense network component in the restoration mapping component, and using the mapping result as the computing power resource usage inference result, wherein the dense network component is used to map the embedded information output by the embedding mapping component into data of the preset dimension.
[0140] When the computer system executes step S103, the dense network component in the restoration mapping component maps the second target representation array into a mapping result of a preset dimension, and uses the mapping result as the computing power resource usage inference result. This process ensures that the high-dimensional feature space is returned to the low-dimensional space of the original computing power resource demand prediction, thereby obtaining a specific computing power resource usage prediction value.
[0141] Specifically, the computer system first obtains the second target representation array corresponding to each first monitoring moment. Assuming that these representation arrays are generated by the embedding mapping component in the previous step, for example, for the monitoring moments from 12:00 to 13:00 on August 10, the second target representation array may be [0.8, 0.75, 1.5, 0.7, 0.2, 0.9, ...], which is a high-dimensional vector processed by the embedding mapping component and contains rich feature information. Next, the computer system processes the second target representation array by restoring the dense network component in the mapping component. The dense network component is usually composed of multiple fully connected layers, and the neurons in each fully connected layer are connected to all neurons in the previous layer. For example, assume that the dense network component contains two fully connected layers, the first fully connected layer has 100 neurons, and the second fully connected layer has 50 neurons. The second target representation array is input to the dense network component, and after passing through the first fully connected layer, a new vector is output, and then after passing through the second fully connected layer, a mapping result of a preset dimension is finally output.
[0142] To illustrate this process, assume that the length of the second target representation array is 20, and the final computing resource usage inference result needs to be a vector of length 6, representing CPU utilization, memory utilization, network traffic, storage usage, number of running tasks, and computing resource demand in the same time period in the past. Then, the structure of the dense network component may be as follows:
[0143] 1. Input layer: accepts a second target representation array of length 20.
[0144] 2. The first fully connected layer: contains 100 neurons and the activation function is ReLU.
[0145] 3. The second fully connected layer: contains 50 neurons and the activation function is ReLU.
[0146] 4. Output layer: contains 6 neurons, and the activation function is a linear function (or an appropriate regression activation function).
[0147] In this process, each fully connected layer uses a weight matrix and a bias vector to perform a linear transformation on the input data, and then introduces nonlinearity through an activation function. For example, the calculation of the first fully connected layer can be expressed as:
[0148] ;
[0149] Among them, h1 is the output of the first fully connected layer, W1 is the weight matrix of the first fully connected layer, b1 is the bias vector of the first fully connected layer, and input is the second target representation array of the input.
[0150] The calculation of the second fully connected layer follows a similar pattern:
[0151] ;
[0152] h2 is the output of the second fully connected layer, W2 is the weight matrix of the second fully connected layer, and b2 is the bias vector of the second fully connected layer.
[0153] The final output layer maps h2 to a vector of length 6, represented as:
[0154] ;
[0155] In this process, W3 and b3 are the weight matrix and bias vector of the last fully connected layer respectively. The final output is the computing resource usage predicted by the computer system, for example [85, 70, 1600, 75, 25, 95], which respectively correspond to the future CPU utilization, memory usage, network traffic, storage usage, number of running tasks and computing resource demand in the same time period in the past.
[0156] In this way, the computer system uses dense network components to effectively restore the second target representation array in the high-dimensional feature space to data of preset dimensions, that is, the computing resource usage reasoning result. This process not only improves the expressiveness of the model, but also ensures the accuracy and reliability of the prediction results, providing a scientific basis for the resource scheduling of the computing platform.
[0157] As an implementation mode, in the step S101, that is, before the past computing resource usage logs corresponding to the first monitoring moments are loaded into the target machine learning network, the method further includes:
[0158] Step S1011: Acquire multiple debugging learning samples and determine a target machine learning network to be debugged, wherein the debugging learning samples include a sample past computing resource usage log at a first sample monitoring time and a test past computing resource usage log at a second sample monitoring time, wherein the second sample monitoring time is a time after the first sample monitoring time;
[0159] Step S1012: Load the multiple debugging learning samples into the target machine learning network to be debugged, and obtain multiple mapping results of the target machine learning network to be debugged, wherein the mapping results are data inferred by the target machine learning network to be debugged based on the past computing resource usage logs of the samples;
[0160] Step S1013: determining a test past computing resource usage log corresponding to each mapping result in the multiple mapping results, and determining the current loss of the target machine learning network to be debugged according to the mapping result and the corresponding test past computing resource usage log;
[0161] Step S1014: if the current loss meets a preset judgment criterion, determining the target machine learning network to be debugged as the target machine learning network;
[0162] Step S1015: If the current loss does not meet the preset judgment criteria, the network parameters of the target machine learning network to be debugged are optimized, and the optimized target machine learning network is determined as the target machine learning network to be debugged.
[0163] Before executing step S101, the computer system debugs and optimizes the target machine learning network through a series of sub-steps to ensure that it can accurately predict future computing resource requirements. This process is divided into five sub-steps: S1011 to S1015.
[0164] In step S1011, the computer system obtains multiple debugging learning samples and determines the target machine learning network to be debugged. These debugging learning samples include two parts: one part is the sample past computing power resource usage log at the first sample monitoring time, and the other part is the test past computing power resource usage log at the second sample monitoring time. The second sample monitoring time is the moment after the first sample monitoring time. For example, the computer system can obtain the computing power resource usage log from 12:00 to 13:00 every day from August 1 to August 7, 2024 as the data of the first sample monitoring time, and at the same time obtain the computing power resource usage log from 12:00 to 13:00 on August 8 as the data of the second sample monitoring time. Specifically, assume that the sample past computing resource usage log at the first sample monitoring time from 12:00 to 13:00 on August 1 is [80, 65, 70, 30, 0, 0], and the test past computing resource usage log at the second sample monitoring time from 12:00 to 13:00 on August 8 is [85, 70,75, 35, 0, 0]. The computer system loads these samples into the target machine learning network to be debugged, which is a neural network model trained by debugging learning samples.
[0165] In step S1012, the computer system loads multiple debugging learning samples into the target machine learning network to be debugged, and obtains multiple mapping results of the target machine learning network to be debugged. These mapping results are data obtained by reasoning based on the sample's past computing resource usage log. Continuing with the data from 12:00 to 13:00 on August 1 as an example, the computer system inputs [80, 65, 70, 30, 0, 0] into the target machine learning network to be debugged, and obtains a mapping result. Assume that the output of the network is a vector of length 6, representing a prediction of computing resource demand at future times. For example, the mapping result of the network output may be [83, 68, 72, 32, 0, 0]. The computer system continues this process until all debugging learning samples have been processed, thereby obtaining a series of mapping results. These mapping results will be used to evaluate the prediction performance of the network.
[0166] In step S1013, the computer system determines the test past computing resource usage log corresponding to each mapping result in the multiple mapping results, and determines the current loss of the target machine learning network to be debugged based on the mapping result and the corresponding test past computing resource usage log. The loss function is used to measure the gap between the predicted result and the actual value. For example, the mean squared error (MSE) can be used as the loss function:
[0167] ;
[0168] Among them, y i Indicates the actual usage of computing resources. represents the network's predicted computing resource usage, and n is the number of samples. Assume that the loss calculated by the computer system using the above formula is 0.5.
[0169] In step S1014, the computer system checks whether the current loss meets the preset judgment criteria. If the loss meets the preset criteria, such as being less than a certain threshold, the computer system determines the current target machine learning network to be debugged as the target machine learning network. This means that the network is accurate enough to be used for actual computing resource demand prediction tasks.
[0170] In step S1015, if the current loss does not meet the preset judgment criteria, the computer system will optimize the network parameters of the target machine learning network to be debugged. This optimization process usually involves optimization algorithms such as gradient descent to adjust the weights and biases in the network so that the loss function value gradually decreases. For example, the computer system can use Stochastic Gradient Descent (SGD) or Adam optimizer to update the network parameters. After the optimization is completed, the computer system determines the optimized target machine learning network as the new target machine learning network to be debugged, and repeats the above process until the loss meets the preset criteria.
[0171] In this way, the computer system can ensure that the target machine learning network has been fully debugged and optimized, so that it has high prediction accuracy and reliability. The final target machine learning network will be used to predict the computing resource demand in the subsequent steps, providing an effective resource scheduling basis for the computing platform.
[0172] As an implementation mode, in the step S1013, the current loss of the target machine learning network to be debugged is determined based on the mapping result and the corresponding test past computing power resource usage log, including: based on a preset plurality of loss functions, the loss of the mapping result and the corresponding test past computing power resource usage log is jointly determined to obtain the current loss.
[0173] When the computer system executes step S1013, based on the mapping results and the corresponding test past computing resource usage logs, the loss is determined by using a preset multiple loss functions to jointly determine the mapping results and the corresponding test past computing resource usage logs, thereby obtaining the current loss. This process ensures a comprehensive evaluation of network performance and improves the prediction accuracy of the model.
[0174] Specifically, the computer system first obtains multiple debugging learning samples and loads these samples into the target machine learning network to be debugged. Through network processing, a series of mapping results are obtained. For example, for the computing power resource usage log from 12:00 to 13:00 every day from August 1 to August 7, 2024, the computer system obtains the corresponding mapping results. Assume that the sample past computing power resource usage log at the first sample monitoring time from 12:00 to 13:00 on August 1 is [80, 65,70, 30, 0, 0], and the test past computing power resource usage log at the second sample monitoring time from 12:00 to 13:00 on August 8 is [85, 70, 75, 35, 0, 0]. The mapping result output by the network may be [83, 68, 72, 32, 0, 0]. Next, the computer system uses a preset multiple loss functions to evaluate the difference between the mapping result and the test past computing power resource usage log. Common loss functions include Mean Squared Error (MSE), Mean Absolute Error (MAE), etc. These loss functions calculate the gap between the mapping results and the actual test data.
[0175] First, the computer system uses mean square error (MSE) to calculate the loss. MSE is a commonly used loss function that is suitable for regression problems. Its calculation formula is:
[0176] ;
[0177] Among them, y i Indicates the actual usage of computing resources. Indicates the computing resource usage predicted by the network, and n is the number of samples. For example, for the test data from 12:00 to 13:00 on August 8, assuming that the actual computing resource usage is [85, 70, 75, 35, 0, 0], and the network prediction result is [83, 68, 72, 32, 0, 0], the MSE is calculated as follows:
[0178] ;
[0179] ;
[0180] Next, the computer system uses the mean absolute error (MAE) to calculate the loss. MAE is also a commonly used loss function, and its calculation formula is:
[0181] ;
[0182] Continuing with the above example, the MAE is calculated as follows:
[0183] ;
[0184] ;
[0185] The computer system considers the results of MSE and MAE and determines the final loss by weighted average or other methods. For example, assuming that the weights of MSE and MAE are 0.6 and 0.4 respectively, the final loss is calculated as follows:
[0186] ;
[0187] In this way, the computer system can more comprehensively evaluate the performance of the target machine learning network to be debugged, ensuring that it can achieve good prediction results under multiple evaluation indicators. The final loss value will be used to determine whether the network meets the preset judgment criteria. If not, the network parameters will be optimized until the loss value reaches the expected level.
[0188] Through this process, the computer system ensures that the target machine learning network is fully debugged and optimized under the joint action of multiple loss functions, thereby improving the accuracy and reliability of computing resource demand prediction.
[0189] As an implementation mode, in the step S100, that is, before determining a plurality of first monitoring moments on the target monitoring data stream and retrieving a past computing resource usage log corresponding to each of the plurality of first monitoring moments, the method further includes:
[0190] Step S10: obtaining a computing power demand estimation command generated for a preset computing power platform, wherein the computing power demand estimation command is used to trigger inference of the computing power demand of the preset computing power platform at the second monitoring time, and the target monitoring data stream is the monitoring data stream of the preset computing power platform;
[0191] The step S100, i.e., determining multiple first monitoring moments on the target monitoring data stream, and retrieving the past computing power resource usage logs corresponding to each of the multiple first monitoring moments, may include: based on the computing power demand estimation command, determining the multiple first monitoring moments on the target monitoring data stream that are associated with the second monitoring moment, and retrieving the past computing power resource usage logs corresponding to each of the first monitoring moments.
[0192] The computer system obtains a computing power demand estimation command generated for a preset computing power platform, which serves as a trigger to start the reasoning process of the computing power demand of the preset computing power platform at the specified second monitoring time. The computing power demand estimation command contains necessary parameters that specify the time window to be analyzed, that is, the second monitoring time, so that the system can determine which past data is associated with the current demand forecast. For example, if the computing power demand estimation command points to the time period from 14:00 to 15:00 on August 25, 2024 as the second monitoring time, the computer system will look for past data points associated with this time period in order to better predict the computing power demand within this time period.
[0193] Subsequently, the computer system determines multiple first monitoring moments on the target monitoring data stream that are associated with the second monitoring moment based on the computing power demand estimation command. This means that the system will look for past records that are closest to the specified time period and select several representative moments from them as the first monitoring moment. For example, the computer system may select the computing power resource usage in the same time period every day (for example, 14:00 to 15:00) from August 1 to August 24, 2024 as the data for the first monitoring moment to construct a monitoring data stream containing these moments. Through such a selection, the system is able to use recent past data with similar patterns to improve the accuracy of the prediction.
[0194] After determining the first monitoring moment, the computer system retrieves the past computing resource usage logs corresponding to each first monitoring moment. This step involves extracting specific computing power usage records from the stored past data. For example, if the computing power usage from 14:00 to 15:00 on August 1 is [80, 65, 70, 30, 0, 0] (each number here represents a different computing power resource usage indicator, such as CPU usage, memory usage, etc.), then this record will be extracted as part of the training data. The system may also extract other similar data records to form a matrix consisting of multiple vectors, where each row represents the data at a first monitoring moment and each column represents the usage of a resource type.
[0195] After completing the above data preparation steps, the computer system inputs this data into a pre-trained machine learning model to infer the computing power requirements at the second monitoring moment. For example, if the model used is a long short-term memory network (LSTM), the system inputs the above data set into the model in time series order, and the LSTM will make predictions about the computing power requirements at future moments based on the patterns and trends in the input sequence. In this way, the computer system can effectively predict future computing power requirements based on past data, thereby helping the computing power platform to better plan and allocate resources.
[0196] As an implementation method, after obtaining the inference result of the computing power resource usage corresponding to the second monitoring time, the operation health of the current computing power platform operation data can also be evaluated, so that the load scheduling strategy can be specified based on the computing power resource usage obtained by reasoning and the current operation health evaluated, so that the computing power platform can reach a normal operation state at the second monitoring time. The following describes the process of evaluating the operation health of the current computing power platform operation data, which can include the following steps:
[0197] Step S60: integrating the multi-channel representation array extracted from the operation log of the computing power platform operation data to be identified, so as to obtain the multi-channel operation representation array corresponding to the computing power platform operation data;
[0198] Step S70: Based on the multi-channel operation characterization array and the first influence coefficient set determined based on the first detection mark, a first health detection parameter set is obtained, wherein the first detection mark is a health detection mark constructed according to the health definition, and the first health detection parameter set represents the confidence level of the computing power platform operation data added to each operation health according to the first detection mark;
[0199] Step S80: Based on the multi-channel operation characterization array and the second influence coefficient set determined based on the second detection mark, a second health detection parameter set is obtained, wherein the second detection mark is a health detection mark constructed according to the alarm coefficient, and the second health detection parameter set represents the confidence level of the computing power platform operation data added to each operation health according to the second detection mark;
[0200] Step S90: Based on the first health detection parameter set and the second health detection parameter set, determine the target operating health that matches the computing power platform operating data.
[0201] After obtaining the inference result of the computing power resource usage corresponding to the second monitoring time, the computer system further evaluates the operation health of the current computing power platform operation data. This process aims to formulate a load scheduling strategy based on the inferred computing power resource usage and the evaluated current operation health, so as to ensure that the computing power platform can reach a normal operating state at the second monitoring time. The evaluation process is divided into four steps: step S60, step S70, step S80 and step S90.
[0202] First, in step S60, the computer system extracts a multi-channel characterization array from the operation log of the computing power platform operation data to be identified, and integrates it to obtain a multi-channel operation characterization array corresponding to the computing power platform operation data. For example, suppose the system extracts operation log data of four different dimensions: CPU utilization, memory usage, network traffic, and disk I / O operations. Each dimension contains data records at multiple times, forming a multi-channel characterization array. The integrated array may be in the form of [[80,75,70], [60,65,70], [500,550,600], [20,25,30]], where each row represents the data of a channel, and each column represents a different combination of channel data at a time.
[0203] Next, in step S70, the computer system obtains a first health detection parameter set based on the multi-channel operation characterization array and the first influence coefficient set determined based on the first detection mark. Here, the first detection mark is a health detection mark constructed according to the health definition, which is used to evaluate the health of the computing power platform. For example, the system can set a threshold matrix, each threshold represents a healthy indicator range, such as CPU utilization is less than 85%, memory usage is less than 70%, etc. The first influence coefficient set is a set of weights obtained by training the machine learning model to quantify the degree of influence of different operating indicators on the overall health. Assuming that the first influence coefficient set is [0.3, 0.2, 0.25, 0.25], the first health detection parameter set can be obtained by multiplying the multi-channel operation characterization array with the first influence coefficient set. For example, if the data at a certain moment is taken, its first health detection parameter is 80×0.3+60×0.2+500×0.25+20×0.25=182.5. In order to ensure that the final confidence level is between 0 and 1, normalization processing can be used. For example, divide the first health detection parameter by the maximum possible value (assumed to be 1000) to obtain the normalized confidence level: normalized confidence level = 182.5 / 1000 = 0.182. In this way, even if the original health detection parameter may be large, the normalized confidence level is always between 0 and 1, ensuring the validity of the result.
[0204] In step S80, the computer system obtains a second health detection parameter set based on the multi-channel operation characterization array and the second influence coefficient set determined based on the second detection mark. Different from the first detection mark, the second detection mark is constructed based on the alarm coefficient and is mainly used to evaluate the alarm situation that occurs in the computing power platform within a specific time period. Assuming that the second influence coefficient set is [0.2, 0.3, 0.3, 0.2], the second health detection parameter set is also obtained by multiplying the multi-channel operation characterization array and the second influence coefficient set.
[0205] Finally, in step S90, the computer system determines the target operating health that matches the computing power platform's operating data based on the first health detection parameter set and the second health detection parameter set. This process usually involves weighted averaging or other comprehensive evaluation methods to fuse the information of the two health detection parameter sets. For example, the computer system can combine the information of the two sets in a linear combination manner, and the formula is: target operating health = α × first health detection parameter + (1-α) × second health detection parameter, where α is a weight factor between 0 and 1, which is used to balance the importance of the two sets. In this way, the computer system can comprehensively evaluate the current operating health of the computing power platform, and adjust the load scheduling strategy accordingly to ensure optimal resource allocation and system stability at the second monitoring time.
[0206] As an implementation manner, the step S70, based on the multi-channel operation characterization array and the first influence coefficient set determined based on the first detection mark, obtains a first health detection parameter set, including:
[0207] Step S71: Obtain a fusion result between the multi-channel operation characterization array and each influence coefficient in the first influence coefficient set to obtain the first health detection parameter set.
[0208] The step S80, based on the multi-channel operation characterization array and the second influence coefficient set determined based on the second detection mark, obtains a second health detection parameter set, including:
[0209] Step S81: Obtain a fusion result between the multi-channel operation characterization array and each influence coefficient in the second influence coefficient set to obtain the second health detection parameter set.
[0210] When the computer system executes steps S70 and S80, based on the multi-channel operation characterization array and the influence coefficient set determined by the corresponding detection mark, the first health detection parameter set and the second health detection parameter set are obtained. This process is divided into two main steps: step S71 and step S81.
[0211] In step S71, the computer system obtains the fusion result between the multi-channel operation characterization array and each influence coefficient in the first influence coefficient set, that is, the result of weighted summation, and obtains the first health detection parameter set. This process involves weighted summation of each channel data in the multi-channel operation characterization array and the corresponding influence coefficient in the first influence coefficient set.
[0212] Specifically, the computer system first extracts a multi-channel representation array from the operation log of the computing platform operation data. Assume that the system extracts operation log data of four different dimensions: CPU utilization, memory usage, network traffic, and disk I / O operations. Each dimension contains data records at multiple times, forming a multi-channel representation array. For example, the multi-channel operation representation array may be in the form of [[80,75,70],[60,65,70],[500,550,600],[20,25,30]], where each row represents the data of a channel, and each column represents a different combination of channel data at a time.
[0213] Next, the computer system obtains a first set of influence coefficients. The first set of influence coefficients is a set of weights obtained through past data analysis and machine learning model training to quantify the degree of influence of different operating indicators on overall health. Assuming that the first set of influence coefficients is [0.3, 0.2, 0.25, 0.25], the first set of health detection parameters can be obtained by multiplying the multi-channel operation characterization array with the first set of influence coefficients.
[0214] For each moment, the computer system performs a weighted summation of the data in the multi-channel operation characterization array and the influence coefficients in the first influence coefficient set. For example, for the data at the first moment, the first health detection parameter is 80×0.3+60×0.2+500×0.25+20×0.25=182.5. Similarly, for the data at the second moment, the first health detection parameter is 75×0.3+65×0.2+550×0.25+25×0.25=185. For the data at the third moment, the first health detection parameter is 70×0.3+70×0.2+600×0.25+30×0.25=187.5.
[0215] In this way, the computer system obtains a set of first health detection parameters for each moment, such as [182.5, 185, 187.5]. This set represents the confidence level added to each operation health by the computing platform operation data according to the first detection mark.
[0216] In step S81, the computer system obtains the fusion result between the multi-channel operation characterization array and each influence coefficient in the second influence coefficient set, that is, the result of weighted summation, to obtain the second health detection parameter set. This process is similar to step S71, but based on different detection tags and influence coefficient sets.
[0217] Specifically, the computer system again uses the previously extracted multi-channel operation characterization array [[80,75,70], [60,65,70], [500,550,600], [20,25,30]]. This time, the system obtains the second set of influence coefficients. The second set of influence coefficients is also a set of weights obtained through past data analysis and machine learning model training, but it focuses on evaluating the alarm conditions that occur in the computing power platform within a specific time period. Assuming that the second set of influence coefficients is [0.2, 0.3, 0.3, 0.2], the second health detection parameter set is also obtained by multiplying the multi-channel operation characterization array by the second set of influence coefficients.
[0218] For the data at the first moment, the second health detection parameter is 80×0.2+60×0.3+500×0.3+20×0.2=200. For the data at the second moment, the second health detection parameter is 75×0.2+65×0.3+550×0.3+25×0.2=205. For the data at the third moment, the second health detection parameter is 70×0.2+70×0.3+600×0.3+30×0.2=210.
[0219] In this way, the computer system obtains a set of second health detection parameters for each moment, such as [200, 205, 210]. This set represents the confidence level added to each operation health by the computing platform operation data according to the second detection mark.
[0220] Through the above steps S71 and S81, the computer system can obtain the first health detection parameter set and the second health detection parameter set respectively based on the multi-channel operation characterization array and the corresponding influence coefficient set. These sets will be used in the subsequent step S90 to determine the target operation health matching the computing power platform operation data, so as to guide the computing power platform to reach a normal operation state at the second monitoring time.
[0221] As an implementation manner, in step S60, before integrating the multi-channel characterization array extracted from the operation log of the computing power platform operation data to be identified to obtain the multi-channel operation characterization array corresponding to the computing power platform operation data, the method further includes:
[0222] Step S601: Obtain a first operation data learning sample set;
[0223] Step S602: assigning the first detection mark to each first operation data learning sample in the first operation data learning sample set according to the health definition;
[0224] Step S603: inputting the first operation data learning sample set and the corresponding first detection mark into the initialized operation health detection network for debugging to obtain a debugging execution result, wherein, in each round of debugging of the operation health detection network, the first sample operation health corresponding to the first operation data learning sample is determined based on the multi-channel representation array extracted from the operation log of the first operation data learning sample;
[0225] Step S604: When the debugging execution result indicates that a first debugging cutoff requirement is met, a target operating health detection network for obtaining the first health detection parameter set is determined, wherein the first debugging cutoff requirement represents that a loss between the determined operating health of the first sample and the operating health indicated by the first detection mark is not greater than a first reference loss.
[0226] Before executing step S60, the computer system integrates the multi-channel characterization array extracted from the operation log of the computing power platform operation data to be identified through a series of sub-steps to obtain a multi-channel operation characterization array corresponding to the computing power platform operation data. This process is divided into four sub-steps: step S601 to step S604.
[0227] In step S601, the computer system obtains a first set of operating data learning samples. These sample sets contain the operating data of the computing power platform at different times, which are used to train the operating health detection network. For example, assume that the computer system collects operating data from 14:00 to 15:00 every day from August 1 to August 24, 2024, to form a first set of operating data learning samples. Each sample contains data from multiple channels, such as CPU utilization, memory utilization, network traffic, disk I / O operations, etc. Assume that the data of each sample is a four-dimensional array, such as [80, 65, 500, 20], indicating that the CPU utilization at a certain moment is 80%, the memory utilization is 65%, the network traffic is 500KB / s, and the disk I / O operation is 20 times / second.
[0228] In step S602, the computer system assigns a first detection mark to each first operation data learning sample in the first operation data learning sample set according to the health definition. The first detection mark is set according to the health definition and is used to indicate the health status of the sample. For example, the computer system can set a health threshold. When the CPU utilization rate is lower than 85%, the memory utilization rate is lower than 70%, the network traffic is lower than 600KB / s, and the disk I / O operation is lower than 30 times / second, it is marked as a healthy state (for example, marked as 1); otherwise, it is marked as an unhealthy state (for example, marked as 0). Assuming that the data from 14:00 to 15:00 on August 1 is [80, 65, 500, 20], the data is marked as a healthy state (marked as 1).
[0229] In step S603, the computer system inputs the first operation data learning sample set and the corresponding health mark into the initialized operation health detection network for debugging to obtain the debugging execution result. In each round of debugging, the operation health of the first sample corresponding to the first operation data learning sample is determined based on the multi-channel representation array extracted from the operation log of the first operation data learning sample. The operation health detection network can be a neural network, such as a convolutional neural network (CNN) or a recurrent neural network (RNN), which is used to learn and predict the operation health from the multi-channel representation array.
[0230] Assume that a computer system uses a simple multi-layer perceptron (MLP) as a health check network. During each round of debugging, the network receives a multi-channel representation array as input and outputs a health prediction value. For example, for the data [80, 65, 500, 20] from 14:00 to 15:00 on August 1, the health prediction value output by the network may be 0.95. The computer system compares this prediction value with the actual health mark (1) and calculates the loss value. Commonly used loss functions are mean square error (MSE) or cross-entropy loss. For example, MSE is used as the loss function.
[0231] In step S604, when the debugging execution result indicates that the first debugging cutoff requirement is met, the computer system determines to obtain a target operation health detection network for obtaining a first health detection parameter set. The first debugging cutoff requirement represents that the loss between the determined first sample operation health and the operation health indicated by the first detection mark is not greater than the first reference loss. For example, if the set first reference loss is 0.05, then when the average loss value of all samples is less than 0.05, the computer system considers that the debugging has met the requirement.
[0232] For example, suppose that during the debugging process, the network loss value gradually decreases and eventually reaches 0.04. At this time, the computer system believes that the debugging has been successful and determines the current operation health detection network as the target operation health detection network. This network will be used in subsequent steps to predict the health of the new computing platform operation data.
[0233] Through the above steps S601 to S604, the computer system can learn the sample set and its health mark based on the first operation data, and finally determine a target operation health detection network that can be used to obtain the first health detection parameter set by debugging the operation health detection network. This network will be used in subsequent steps to perform health evaluation on the current computing power platform operation data to ensure that the computing power platform can reach a normal operating state at any time in the future.
[0234] As an implementation manner, in step S60, before integrating the multi-channel characterization array extracted from the operation log of the computing power platform operation data to be identified to obtain the multi-channel operation characterization array corresponding to the computing power platform operation data, the following may also be included:
[0235] Step S60a: Obtain a second operation data learning sample set;
[0236] Step S60b: assigning the second detection mark to each second operation data learning sample in the second operation data learning sample set according to the warning coefficient;
[0237] Step S60c: inputting the second operation data learning sample set and the corresponding second detection mark into the initialized alarm health detection network for debugging, and obtaining a debugging execution result, wherein, in each round of debugging of the alarm health detection network, based on the multi-channel representation array extracted from the operation log of the second operation data learning sample and the alarm coefficient corresponding to the second operation data learning sample, the second sample operation health corresponding to the second operation data learning sample is determined;
[0238] Step S60d: When the debugging execution result indicates that a second debugging cutoff requirement is met, a target alarm health detection network for obtaining the second health detection parameter set is determined, wherein the second debugging cutoff requirement represents that a loss between the determined operating health of the second sample and the operating health indicated by the second detection mark is not greater than a second reference loss.
[0239] Before executing step S60, the computer system integrates the multi-channel characterization array extracted from the operation log of the computing power platform operation data to be identified through a series of sub-steps to obtain a multi-channel operation characterization array corresponding to the computing power platform operation data. This process is divided into four sub-steps: step S60a to step S60d.
[0240] In step S60a, the computer system obtains a second set of operating data learning samples. These sample sets contain the operating data of the computing power platform at different times, which are used to train the alarm health detection network. For example, assume that the computer system collects operating data from 14:00 to 15:00 every day from August 1 to August 24, 2024, to form a second set of operating data learning samples. Each sample contains data from multiple channels, such as CPU utilization, memory utilization, network traffic, disk I / O operations, etc. Assume that the data of each sample is a four-dimensional array, such as [80, 65, 500, 20], indicating that the CPU utilization at a certain moment is 80%, the memory utilization is 65%, the network traffic is 500KB / s, and the disk I / O operation is 20 times / second.
[0241] In step S60b, the computer system assigns a second detection mark to each second operation data learning sample in the second operation data learning sample set according to the alarm coefficient. The alarm coefficient is a parameter calculated based on past alarm data, which is used to indicate the alarm status of the sample. For example, the computer system can calculate an alarm coefficient based on factors such as the number of alarms and alarm levels that occurred in the past period of time. Assuming that the data from 14:00 to 15:00 on August 1 is [80, 65,500, 20], and two minor alarm events occurred during this period, then it can be assigned an alarm coefficient, such as 0.7.
[0242] In step S60c, the computer system inputs the second operation data learning sample set and the corresponding alarm coefficient into the initialized alarm health detection network for debugging to obtain the debugging execution result. In each round of debugging, the second sample operation health corresponding to the second operation data learning sample is determined based on the multi-channel representation array extracted from the operation log of the second operation data learning sample and the alarm coefficient corresponding to the second operation data learning sample. The alarm health detection network can be a neural network, such as a convolutional neural network (CNN) or a recurrent neural network (RNN), which is used to learn and predict the operation health from the multi-channel representation array.
[0243] Assume that a computer system uses a simple multi-layer perceptron (MLP) as an alarm health detection network. During each round of debugging, the network will receive a multi-channel representation array as input and output a health prediction value. For example, for the data [80, 65, 500, 20] from 14:00 to 15:00 on August 1, the health prediction value output by the network may be 0.95. The computer system compares this prediction value with the actual alarm coefficient (0.7) and calculates the loss value. Commonly used loss functions are mean square error (MSE) or cross-entropy loss (Cross-Entropy Loss). For example, use MSE as the loss function.
[0244] In step S60d, when the debugging execution result indicates that the second debugging cutoff requirement is met, the computer system determines to obtain a target alarm health detection network for obtaining a second health detection parameter set. The second debugging cutoff requirement represents that the loss between the determined second sample running health and the running health indicated by the second detection mark is not greater than the second reference loss. For example, if the set second reference loss is 0.05, then when the average loss value of all samples is less than 0.05, the computer system considers that the debugging has met the requirement.
[0245] For example, suppose that during the debugging process, the network loss value gradually decreases and eventually reaches 0.04. At this time, the computer system believes that the debugging has been successful and determines the current alarm health detection network as the target alarm health detection network. This network will be used to predict the health of the new computing platform operation data in the subsequent steps.
[0246] Through the above steps S60a to S60d, the computer system can learn the sample set and its alarm coefficient based on the second operation data, and finally determine a target alarm health detection network that can be used to obtain the second health detection parameter set by debugging the alarm health detection network. This network will be used in subsequent steps to perform health evaluation on the current computing power platform operation data to ensure that the computing power platform can reach a normal operating state at any time in the future.
[0247] Specifically, the computer system first obtains the second operation data learning sample set. Assume that the sample set contains 20 days of data, with a time period from 14:00 to 15:00 every day, and the data of each time period is a four-dimensional array. For example, the data from 14:00 to 15:00 on August 1 is [80, 65, 500, 20], indicating that the CPU utilization rate during this period is 80%, the memory utilization rate is 65%, the network traffic is 500KB / s, and the disk I / O operation is 20 times / second. At the same time, assume that two minor alarm events occurred during this period, so an alarm coefficient of 0.7 is assigned to it.
[0248] Next, the computer system assigns a corresponding alarm coefficient to each sample. Assume that the data from 14:00 to 15:00 on August 2 is [85, 70, 600, 30], and no alarm events occurred during this period, so the alarm coefficient is 0.0. The data from 14:00 to 15:00 on August 3 is [90, 75, 700, 40], and a serious alarm event occurred during this period, so the alarm coefficient is 0.9.
[0249] The computer system inputs these samples and their warning coefficients into the initialized warning health detection network for debugging. In each round of debugging, the network will receive a multi-channel representation array as input and output a health prediction value. For example, for the data [80, 65, 500, 20] from 14:00 to 15:00 on August 1, the health prediction value output by the network may be 0.95. The computer system compares this prediction value with the actual warning coefficient (0.7) and calculates the loss value. Assume that MSE is used as the loss function.
[0250] During the debugging process, the computer system continuously adjusts the network parameters to gradually reduce the loss value. Assume that during the debugging process, the network loss value gradually decreases and eventually reaches 0.04. At this time, the computer system believes that the debugging has been successful and determines the current alarm health detection network as the target alarm health detection network. This network will be used in subsequent steps to predict the health of the new computing platform operation data.
[0251] Through the above steps, the computer system can learn the sample set and its alarm coefficient based on the second operation data, and finally determine a target alarm health detection network that can be used to obtain the second health detection parameter set by debugging the alarm health detection network. This network will be used in subsequent steps to perform health evaluation on the current computing power platform operation data to ensure that the computing power platform can reach a normal operating state at any time in the future.
[0252] Figure 2 A hardware entity diagram of a computer system provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the hardware entity of the computer system 1000 includes: a processor 1001 and a memory 1002, wherein the memory 1002 stores a computer program that can be run on the processor 1001, and the processor 1001 implements the steps in the method of any of the above embodiments when executing the program.
[0253] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.
Claims
1. A method for predicting resource demand of a computing power platform, characterized in that: Applied to a computer system, the method comprises: Determine a plurality of first monitoring moments on a target monitoring data stream, and retrieve a past computing resource usage log corresponding to each of the plurality of first monitoring moments; According to the past computing resource usage logs corresponding to the first monitoring moments, obtaining the past computing resource representation arrays corresponding to the first monitoring moments; When the timing representation arrays corresponding to the first monitoring moments are obtained, the past computing resource representation arrays and timing representation arrays corresponding to the first monitoring moments are respectively integrated to obtain the first target representation arrays corresponding to the first monitoring moments, wherein the timing representation arrays represent the timing characteristics of the first monitoring moment on the target monitoring data stream; According to the first target representation array, obtaining a second target representation array corresponding to the plurality of first monitoring moments, wherein the second target representation array represents the correlation between the past computing resource usage logs corresponding to the respective first monitoring moments; Based on the second target representation array, reasoning on a second monitoring time on the target monitoring data stream to obtain a computing power resource usage reasoning result corresponding to the second monitoring time, wherein the second monitoring time is a time after the first monitoring time; The step of obtaining the second target characterization array corresponding to the plurality of first monitoring moments according to the first target characterization array includes: When one or more linear transformation arrays are obtained, feature mapping is performed on the first target representation arrays corresponding to the respective first monitoring moments through the one or more linear transformation arrays to obtain second target representation arrays corresponding to the multiple first monitoring moments.
2. The method according to claim 1, characterized in that: When the one or more linear transformation arrays are obtained, feature mapping is performed on the first target representation arrays corresponding to the first monitoring moments by using the one or more linear transformation arrays to obtain the second target representation arrays corresponding to the multiple first monitoring moments, including: When obtaining the first linear transformation array and the second linear transformation array, feature mapping is performed on the first target characterization array corresponding to each first monitoring moment through the first linear transformation array and the second linear transformation array, respectively, to obtain the first feature mapping array and the second feature mapping array corresponding to each first monitoring moment, wherein the first feature mapping array is a characterization array obtained by feature mapping the first target characterization array corresponding to each first monitoring moment through the first linear transformation array, and the second feature mapping array is a characterization array obtained by feature mapping the first target characterization array corresponding to each first monitoring moment through the second linear transformation array; Respectively performing a combination operation on the first feature mapping arrays corresponding to the first monitoring moments and the second feature mapping arrays corresponding to the first monitoring moments to obtain a first linear transformation array corresponding to the first feature mapping array and a second linear transformation array corresponding to the second feature mapping array; The first linear transformation array and the second linear transformation array are processed simultaneously to obtain a second target representation array corresponding to the multiple first monitoring moments.
3. The method according to claim 1, characterized in that The obtaining, according to the past computing resource usage logs corresponding to the first monitoring moments, past computing resource representation arrays corresponding to the first monitoring moments includes: Obtaining the impact factor data corresponding to the past computing resource usage logs corresponding to the first monitoring moments, wherein the impact factor data is dynamically changing control data associated with the past computing resource usage logs; Performing a combination operation on the past computing resource usage logs corresponding to each of the first monitoring moments and the corresponding influencing factor data, to obtain a past computing resource representation array corresponding to each of the first monitoring moments; Before respectively fusing the past computing resource representation arrays and the timing representation arrays corresponding to the first monitoring moments to obtain the first target representation arrays corresponding to the first monitoring moments, the method further includes: The timing characteristics of each first monitoring moment are embedded through wave functions of different wavelengths to obtain a timing representation array corresponding to each first monitoring moment.
4. The method according to claim 1, characterized in that: After determining a plurality of first monitoring moments on the target monitoring data stream, and retrieving a past computing resource usage log corresponding to each of the plurality of first monitoring moments, the method further includes: Loading the past computing power resource usage logs corresponding to each of the first monitoring moments into a target machine learning network, wherein the target machine learning network is a neural network obtained by debugging a learning example, and the target machine learning network is used to complete computing power resource demand reasoning, and the target machine learning network includes an embedding mapping component and a restoring mapping component; Embedding the past computing resource usage logs corresponding to each of the first monitoring moments by the embedding mapping component to obtain the second target representation array; The second target representation array is restored and mapped by the restoration mapping component to obtain the computing power resource usage inference result.
5. The method according to claim 4, characterized in that The performing restoration mapping on the second target representation array by the restoration mapping component to obtain the computing power resource usage inference result includes: The second target representation array is mapped into a mapping result of a preset dimension through a dense network component in the restoration mapping component, and the mapping result is used as the computing power resource usage inference result, wherein the dense network component is used to map the embedded information output by the embedding mapping component into data of the preset dimension.
6. The method according to claim 5, characterized in that Before loading the past computing resource usage logs corresponding to each first monitoring moment into the target machine learning network, the method further includes: Acquire multiple debugging learning samples and determine a target machine learning network to be debugged, wherein the debugging learning samples include a sample past computing resource usage log at a first sample monitoring time and a test past computing resource usage log at a second sample monitoring time, wherein the second sample monitoring time is a time after the first sample monitoring time; Loading the multiple debugging learning samples into the target machine learning network to be debugged, and obtaining multiple mapping results of the target machine learning network to be debugged, wherein the mapping results are data inferred by the target machine learning network to be debugged based on the past computing resource usage logs of the samples; Determine a test past computing resource usage log corresponding to each mapping result in the multiple mapping results, and determine the current loss of the target machine learning network to be debugged based on the mapping result and the corresponding test past computing resource usage log; If the current loss meets the preset judgment criteria, the target machine learning network to be debugged is determined as the target machine learning network; If the current loss does not meet the preset judgment criteria, the network parameters of the target machine learning network to be debugged are optimized, and the optimized target machine learning network is determined as the target machine learning network to be debugged.
7. The method according to claim 6, characterized in that The determining the current loss of the target machine learning network to be debugged according to the mapping result and the corresponding test past computing resource usage log includes: Based on the preset multiple loss functions, the loss of the mapping result and the corresponding test past computing resource usage log is jointly determined to obtain the current loss.
8. The method according to any one of claims 1 to 7, characterized in that: Before determining a plurality of first monitoring moments on the target monitoring data stream and retrieving a past computing resource usage log corresponding to each of the plurality of first monitoring moments, the method further includes: Obtaining a computing power demand estimation command generated for a preset computing power platform, wherein the computing power demand estimation command is used to trigger inference of the computing power demand of the preset computing power platform at the second monitoring time, and the target monitoring data stream is the monitoring data stream of the preset computing power platform; The determining of a plurality of first monitoring moments on the target monitoring data stream, and retrieving a past computing resource usage log corresponding to each of the plurality of first monitoring moments, includes: Based on the computing power demand estimation command, the multiple first monitoring moments associated with the second monitoring moment on the target monitoring data stream are determined, and the past computing power resource usage logs corresponding to the respective first monitoring moments are retrieved.
9. A computer system comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, wherein: When the processor executes the program, the steps in the method according to any one of claims 1 to 8 are implemented.
