An intelligent prediction method and system based on big data analysis

Abstract

This invention belongs to the field of data processing and analysis technology. It provides an intelligent prediction method and system based on big data analysis, comprising: performing overlap and correlation analysis on a database of instantaneous resource spike periods and a database of abnormal traffic inflow periods to determine whether the instantaneous resource spikes in the computer system are caused by abnormal traffic inflows; if so, comparing the overlap and non-overlapping periods obtained from the overlap analysis to determine whether the instantaneous resource spike data of the computer system during its current running period needs to be updated, thereby updating the initial monitoring frequency; monitoring the instantaneous resource spike data and traffic data of the computer system according to the updated initial monitoring frequency; and using an LSTM model to predict instantaneous resource spikes in future periods of the computer system, thereby achieving accurate capture of instantaneous resource spikes by the computer system and improving the accuracy of instantaneous resource spike prediction.

Description

Technical Field

[0001] This invention belongs to the field of data processing and analysis technology, specifically an intelligent prediction method and system based on big data analysis. Background Technology

[0002] To effectively monitor the resource usage of computer systems, a common approach is to set an initial monitoring frequency. However, existing monitoring technologies have many shortcomings, limiting the accurate detection and effective prediction of transient resource spikes.

[0003] Regarding monitoring frequency settings, initial monitoring frequencies are often pre-set based on historical experience or general standards, lacking dynamic adaptability to the real-time operating status of the system. A fixed initial monitoring frequency is difficult to meet the monitoring needs of the system under different operating scenarios. When the monitoring frequency is too low, the system may not be able to detect the occurrence of instantaneous resource spikes in a timely manner, resulting in a delay in problem detection.

[0004] In terms of data correlation analysis, existing monitoring technologies mostly process instantaneous resource spike data and traffic data in isolation, ignoring the potentially close relationship between the two. In fact, the influx of abnormal traffic is often one of the important triggers for instantaneous resource spikes. However, existing monitoring methods lack the ability to perform in-depth fusion analysis of traffic data and resource spike data, making it impossible to accurately determine whether resource spikes are caused by abnormal traffic, let alone predict future resource spike situations based on this correlation.

[0005] Regarding data updates and predictions, prediction models built on old data cannot accurately reflect the current operating characteristics of the system. Due to the lack of timely and accurate updates of instantaneous resource spike data, existing prediction methods are unable to adapt to changes in the system's operating state, resulting in low accuracy and reliability of prediction results, which increases the risk of system failures and performance problems.

[0006] Therefore, the present invention provides an intelligent prediction method and system based on big data analysis. Summary of the Invention

[0007] In order to overcome the shortcomings of the prior art, at least one technical problem raised in the background art is solved.

[0008] The technical solution adopted by this invention to solve its technical problem is: an intelligent prediction method based on big data analysis, comprising:

[0009] S1: By performing time period marking processing on the instantaneous resource spike data obtained by monitoring the computer system during its running period according to the initial monitoring frequency, an instantaneous resource spike time period library is obtained;

[0010] The instantaneous resource spike data represents the duration of the instantaneous resource spike phenomenon caused by the computer system's resource request volume exceeding the preset standard request volume.

[0011] S2: Perform anomaly identification and time period marking on the traffic data of the computer system during its running period to obtain a database of abnormal traffic inflow time periods;

[0012] S3: Perform overlap and correlation analysis on the instantaneous resource spike time period database and the abnormal traffic inflow time period database to determine whether the instantaneous resource spike phenomenon in the computer system is caused by abnormal traffic inflow.

[0013] S4: If so, compare the overlapping and non-overlapping periods obtained by the overlap analysis of the instantaneous resource spike period database and the abnormal traffic inflow period database to determine whether the instantaneous resource spike data needs to be updated. If so, proceed to S5; otherwise, proceed to S6.

[0014] S5: Update the instantaneous resource spike data, and proceed to S6 after the update;

[0015] S6: Update the initial monitoring frequency based on instantaneous resource spike data;

[0016] S7: Monitor the instantaneous resource spikes and traffic data of the computer system according to the updated initial monitoring frequency, and predict the instantaneous resource spikes of the computer system in future time periods by combining the LSTM model.

[0017] Preferably, the instantaneous resource spike time period library is obtained in the following way:

[0018] Based on the start and end times of each instance of instantaneous resource spikes, the time periods of instantaneous resource spikes are determined and compiled into a database of instantaneous resource spike time periods.

[0019] Preferably, the traffic data represents the amount of data transmitted by the computer system in real time per unit time.

[0020] The method for obtaining the database of abnormal traffic inflow periods is as follows:

[0021] Obtain the data transmission volume at different unit time points during the normal operation period of historical traffic, and calculate the mean and standard deviation to obtain the mean and standard deviation of traffic during the normal operation period of historical traffic.

[0022] use The principle is to calculate the traffic anomaly score based on the real-time data transmission volume of the computer system at different unit times during the running period, the average traffic volume during the historical normal operation period, and the standard deviation of traffic volume.

[0023] If the traffic anomaly score does not meet the requirements, it indicates a traffic anomaly. The traffic anomaly influx period is determined based on the start and end times of the traffic anomaly, and the traffic anomaly influx period database is compiled.

[0024] Preferably, the overlap analysis process is as follows:

[0025] If any abnormal traffic inflow period in the abnormal traffic inflow period database is the same as the period corresponding to the instantaneous resource spike period, then the instantaneous resource spike period is marked as an overlapping period; otherwise, it is marked as a non-overlapping period.

[0026] Preferably, the correlation analysis process is as follows:

[0027] Obtain the resource request volume and data transmission volume within different units of time during overlapping periods, and integrate them according to time sequence to obtain the resource request volume sequence and data transmission volume sequence;

[0028] The Pearson correlation coefficient method was used to calculate the resource request sequence and the data transmission sequence;

[0029] If the Pearson correlation coefficient is greater than or equal to the Pearson correlation coefficient threshold, the overlapping period is marked as the period of influence.

[0030] Preferably, the process of determining whether the transient resource spikes in the computer system are caused by abnormal traffic inflows is as follows:

[0031] The proportion of overlapping periods within the instantaneous resource spike period is used to obtain the first impact value;

[0032] The proportion of the number of periods with statistical impact within the overlapping periods is used to obtain the second impact value;

[0033] The sum of the first and second impact values ​​is used to obtain the impact value of the inflow.

[0034] If the impact value of the traffic influx meets the requirements, it means that the instantaneous resource spikes in the computer system are caused by abnormal traffic influx.

[0035] Preferably, the process of comparing overlapping and non-overlapping time periods is as follows:

[0036] Among all overlapping time periods, the overlapping time period with the shortest duration is selected as the reference time period;

[0037] If the duration of any non-overlapping time period is shorter than the duration of the reference time period, it indicates that the instantaneous resource spike data of the computer system during the running period needs to be updated.

[0038] If the duration of all non-overlapping time periods is longer than the duration of the reference time period, it means that there is no need to update the instantaneous resource spike data of the computer system during the running period.

[0039] Preferably, the process of updating the instantaneous resource spike data is as follows:

[0040] The time length corresponding to each non-overlapping period is used as the new instantaneous resource spike data to complete the update of instantaneous resource spike data;

[0041] The process of updating the initial monitoring frequency is as follows:

[0042] Extract the time length corresponding to the shortest instantaneous resource spike period as the new monitoring frequency to complete the update of the initial monitoring frequency.

[0043] Preferably, the process of predicting instantaneous resource spikes in the computer system over future time periods is as follows:

[0044] S71 performs real-time monitoring of instantaneous resource spike data and traffic data of the computer system according to the updated initial monitoring frequency, and converts them into sequence samples in LSTM input format, wherein the sequence samples include spike duration sequence samples and traffic data samples;

[0045] S72, set the historical window k as the input feature and the prediction window d as the output target;

[0046] S73, slide a window over the sequence samples in chronological order to cut the continuous instantaneous resource spike data and flow data into input feature samples and output feature samples. The input feature samples include the spike duration sequence and flow sequence within the historical window k, and the output feature samples include the spike duration sequence within the prediction window d.

[0047] S74, construct an LSTM model, including an input layer, an LSTM layer, and a fully connected layer;

[0048] The input layer is used to receive input feature samples and output feature samples;

[0049] An LSTM layer contains 1-2 stacked LSTM cells;

[0050] The fully connected layer maps the LSTM output to the prediction dimension through the Dense layer;

[0051] The LSTM model is trained and tested by input feature samples and output feature samples to obtain a trained LSTM model.

[0052] S75 acquires the latest glitch duration and flow data in real time, updates the input feature samples, and inputs them into the LSTM model to predict the glitch duration in future periods.

[0053] An intelligent prediction system based on big data analytics includes:

[0054] Instantaneous resource glitch marking module: By marking the instantaneous resource glitch data obtained from monitoring the computer system during its running period according to the initial monitoring frequency, a database of instantaneous resource glitch time periods is obtained;

[0055] Traffic anomaly inflow marking module: performs anomaly identification and time period marking on the traffic data of the computer system during the running period, and obtains a traffic anomaly inflow time period library;

[0056] The correlation analysis module performs overlap and correlation analysis on the instantaneous resource spike time period database and the abnormal traffic inflow time period database to determine whether the instantaneous resource spike phenomenon in the computer system is caused by abnormal traffic inflow.

[0057] Data update judgment module: If yes, compare the overlapping and non-overlapping time periods obtained by the overlap analysis of the instantaneous resource spike time period library and the abnormal traffic inflow time period library to determine whether the instantaneous resource spike data needs to be updated. If it needs to be updated, enter the spike data update module; if it does not need to be updated, enter the monitoring and adjustment update module.

[0058] The glitch data update module updates the instantaneous resource glitch data and then enters the monitoring, adjustment and update module after the update.

[0059] Monitoring, adjustment and update module: Updates the initial monitoring frequency based on instantaneous resource spike data;

[0060] Instantaneous resource spike prediction module: Monitors instantaneous resource spike data and traffic data of the computer system according to the updated initial monitoring frequency, and predicts instantaneous resource spikes of the computer system in future time periods by combining the LSTM model.

[0061] The beneficial effects of this invention are as follows: By performing time-period marking processing on the instantaneous resource spike data and traffic data obtained from monitoring the computer system during its running period according to the initial monitoring frequency, a database of instantaneous resource spike time periods and a database of abnormal traffic inflow time periods are obtained. Overlap analysis and correlation analysis are then performed to determine whether the instantaneous resource spike phenomenon in the computer system is caused by abnormal traffic inflow. If so, the overlapping and non-overlapping time periods obtained from the overlap analysis of the instantaneous resource spike time period database and the database of abnormal traffic inflow time periods are compared to determine whether it is necessary to monitor the computer system during its running period. This invention updates instantaneous resource spike data and, based on this data, updates the initial monitoring frequency. It then monitors the computer system's instantaneous resource spike data and traffic data according to the updated initial monitoring frequency. Finally, it uses an LSTM model to predict instantaneous resource spikes in the computer system's future timeframe. The invention primarily addresses the issue of updating the initial monitoring frequency of the computer system to prevent it from failing to accurately capture or monitor instantaneous resource spikes. After updating the initial monitoring frequency, it combines the instantaneous resource spike data and traffic data monitored according to the updated initial monitoring frequency with an LSTM model to predict instantaneous resource spikes in the computer system's future timeframe, thereby improving the accuracy of instantaneous resource spike prediction. Attached Figure Description

[0062] The invention will now be further described with reference to the accompanying drawings.

[0063] Figure 1 This is a flowchart illustrating the steps of an intelligent prediction method based on big data analysis as described in an embodiment of the present invention.

[0064] Figure 2 This is a flowchart of an intelligent prediction system based on big data analysis, as described in an embodiment of the present invention. Detailed Implementation

[0065] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0066] This invention provides an intelligent prediction method and system based on big data analysis. Its main application addresses the inability of traditional monitoring to accurately detect instantaneous resource spikes in computer systems. Specifically, traditional monitoring has a large sampling interval (e.g., minutes), while instantaneous resource spikes in computer systems have short-duration, high-frequency characteristics, making it impossible to accurately detect these spikes. This leads to inaccurate predictions of subsequent instantaneous resource spikes, which is detrimental to the safe and stable operation of the computer system. Instantaneous resource spikes refer to high-frequency, high-intensity peak resource requests or consumption that occur within a very short time (usually measured in seconds or milliseconds). For example, a computer system with a processing capacity of 5000 API calls per second might suddenly experience 10000 requests per second at a certain moment.

[0067] Example 1: Please refer to Figure 1 As shown in the embodiment of the present invention, an intelligent prediction method based on big data analysis includes the following steps:

[0068] S1: By performing time period marking processing on the instantaneous resource spike data obtained by monitoring the computer system during its running period according to the initial monitoring frequency, an instantaneous resource spike time period library is obtained;

[0069] In S1, instantaneous resource spike data represents the duration of an instantaneous resource spike caused by the computer system's resource request volume exceeding the preset standard request volume. For example, if the computer system has a processing capacity of 5,000 API calls per second, and at a certain moment there is a sudden surge of 10,000 requests per second, lasting for 5 seconds.

[0070] In S1, the process of time-period marking for instantaneous resource spike data is as follows:

[0071] Based on the instantaneous resource spike data, determine the duration of each instantaneous resource spike in the computer system, and determine the time period of each instantaneous resource spike in the computer system based on the start and end times corresponding to the duration. That is, the instantaneous resource spike time period. All instantaneous resource spike time periods are summarized to obtain the instantaneous resource spike time period library.

[0072] S2: Perform anomaly identification on the traffic data of the computer system during the running period, and perform time period marking processing after anomaly identification to obtain the traffic anomaly influx time period database;

[0073] In S2, traffic data represents the amount of data transmitted by the computer system in real time per unit time.

[0074] In S2, the process of identifying traffic data of the computer system during its running period is as follows:

[0075] Based on the historical operation logs of the computer system, the data transmission volume at different unit time points during the normal operation period of the historical traffic is extracted, and the mean and standard deviation are calculated to obtain the mean and standard deviation of traffic during the normal operation period of the historical traffic.

[0076] It should be noted that the historical normal traffic period means that the amount of data transmitted by the computer system at different points in time during that period is within the normal traffic standard range;

[0077] use The principle is to calculate the traffic anomaly score Yy based on the real-time data transmission volume, historical average traffic volume during normal operation periods, and traffic standard deviation of the computer system within different unit time intervals during the running period. The specific calculation formula is as follows: ;

[0078] Where YL represents the real-time data transmission volume of the computer system within different unit time periods during the running period, LJ is the average traffic value, and LB is the standard deviation of traffic value.

[0079] Compare the traffic anomaly score Yy with the traffic anomaly score threshold;

[0080] If the abnormal traffic score Yy is greater than or equal to the abnormal traffic score threshold, it indicates that the amount of data transmitted by the computer system per unit time is abnormal, i.e., abnormal traffic.

[0081] If the traffic anomaly score Yy is less than the traffic anomaly score threshold, it means that the computer system's data transmission volume per unit time is normal, i.e., the traffic is normal.

[0082] In S2, the time period marked after anomaly identification of traffic data is as follows:

[0083] During the running period of the computer system, the start and end times of each traffic anomaly are marked. The time period between the start and end times of the traffic anomaly is the traffic anomaly influx period. All traffic anomaly influx periods are summarized to obtain the traffic anomaly influx period library.

[0084] S3: Perform overlap and correlation analysis on the instantaneous resource spike time period database and the abnormal traffic inflow time period database to determine whether the instantaneous resource spike phenomenon in the computer system is caused by abnormal traffic inflow.

[0085] In S3, the process of performing overlap analysis on the instantaneous resource spike time period database and the abnormal traffic inflow time period database is as follows:

[0086] Based on any instantaneous resource spike time period in the instantaneous resource spike time period library;

[0087] If any abnormal traffic inflow period in the abnormal traffic inflow period database is the same as the period corresponding to the instantaneous resource spike period, then the instantaneous resource spike period is marked as an overlapping period.

[0088] Understandably, the purpose of obtaining overlapping time periods is that the overlapping time periods represent the same time periods as the abnormal traffic influx and the instantaneous resource spikes. If the time periods corresponding to the abnormal traffic influx and the instantaneous resource spikes are the same, it means that when the computer system experiences an abnormal traffic influx, an instantaneous resource spike also occurs. If there are many overlapping time periods, it means that the computer system is more likely to be affected by the abnormal traffic influx, leading to an instantaneous resource spike.

[0089] If no abnormal traffic inflow period in the abnormal traffic inflow period database corresponds to the same period as the instantaneous resource spike period, then the instantaneous resource spike period will be marked as a non-overlapping period.

[0090] In S3, the process of performing correlation analysis on the instantaneous resource spike time period library and the abnormal traffic inflow time period library is as follows:

[0091] Based on any overlapping time period, obtain the resource request volume and data transmission volume within different units of time during the overlapping time period, and integrate them according to the time sequence to obtain the resource request volume sequence and data transmission volume sequence.

[0092] The Pearson correlation coefficient method was used to calculate the Pearson correlation coefficient between the resource request sequence and the data transmission sequence.

[0093] Compare the Pearson correlation coefficient with the Pearson correlation coefficient threshold;

[0094] If the Pearson correlation coefficient is greater than or equal to the Pearson correlation coefficient threshold, the overlapping period is marked as the period of influence.

[0095] If the Pearson correlation coefficient is less than the Pearson correlation coefficient threshold, the overlapping period is marked as a non-influencing period.

[0096] Understandably, the purpose of obtaining the impact period is to show that there is a linear correlation between abnormal inflow traffic and resource request volume within the overlapping period. For example, if the increase in abnormal inflow traffic leads to an increase in resource request volume, it indicates that there is a correlation between abnormal inflow traffic and resource request volume. This means that abnormal inflow of traffic can affect the generation of instantaneous resource spikes. The more impact periods there are, the greater the possibility that the computer system is affected by abnormal inflow of traffic, leading to instantaneous resource spikes.

[0097] In S3, the process of determining whether a transient resource spike in a computer system is caused by an abnormal influx of traffic is as follows:

[0098] The proportion of overlapping periods within the instantaneous resource spike period is used to obtain the first impact value;

[0099] The proportion of the number of periods with statistical impact within the overlapping periods is used to obtain the second impact value;

[0100] The sum of the first and second impact values ​​is used to obtain the impact value of the inflow.

[0101] Understandably, the first impact value reflects the proportion of overlapping time periods, while the second impact value reflects the proportion of affected time periods. The higher the proportion of both, the greater the likelihood that the computer system is affected by abnormal traffic inflows, leading to instantaneous resource spikes.

[0102] Preferably, the impact value of the inflow is compared with the impact threshold of the inflow;

[0103] If the impact value of the traffic influx is greater than or equal to the impact threshold of the traffic influx, it means that the instantaneous resource spikes in the computer system are caused by abnormal traffic influx.

[0104] If the impact value of the traffic influx is less than the impact threshold of the traffic influx, it means that the instantaneous resource spikes in the computer system are not caused by abnormal traffic influx.

[0105] S4: If yes, compare the overlapping and non-overlapping periods obtained by the overlap analysis of the instantaneous resource spike period database and the abnormal traffic inflow period database to determine whether it is necessary to update the instantaneous resource spike data of the computer system during the running period. If it is necessary, proceed to S5; if not, proceed to S6.

[0106] In S4, the process of comparing overlapping and non-overlapping time periods is as follows:

[0107] Among all overlapping time periods, the overlapping time period with the shortest duration is selected as the reference time period, based on any non-overlapping time period;

[0108] If the duration of any non-overlapping time period is shorter than the duration of the reference time period, it indicates that the instantaneous resource spike data of the computer system during the running period needs to be updated.

[0109] If the duration of all non-overlapping time periods is longer than the duration of the reference time period, it means that there is no need to update the instantaneous resource spike data of the computer system during the running period.

[0110] Understandably, the significance of selecting the shortest overlapping period as the reference period lies in the fact that the initial monitoring frequency of the computer system needs to be updated and optimized subsequently. To more accurately capture or detect instantaneous resource spikes in the computer system, the updated initial monitoring frequency must be less than or equal to the duration of the instantaneous resource spike. Since instantaneous resource spikes have already occurred within the overlapping period, the shortest overlapping period is selected as the reference period. In this application, the purpose of comparing the non-overlapping period with the reference period is that although the initial monitoring frequency setting may have prevented the capture or detection of instantaneous resource spikes in the non-overlapping period, the non-overlapping period... If an anomaly in data transmission occurs during a period of overlap (abnormal traffic data), this anomaly can cause momentary resource spikes in the computer system. This indicates that momentary resource spikes also occur during non-overlapping periods, but were not detected or captured due to the initial monitoring frequency setting. Therefore, to more accurately capture or monitor momentary resource spikes in the computer system, the non-overlapping periods are compared with reference periods. If the duration of a non-overlapping period is shorter than the reference period, it indicates that a shorter momentary resource spike has occurred in the computer system. Therefore, the momentary resource spike data needs to be updated to more accurately update the initial monitoring frequency of the computer system in the future.

[0111] S5: Update the instantaneous resource spike data, and proceed to S6 after the update;

[0112] In S5, the process of updating instantaneous resource spike data is as follows:

[0113] The time length corresponding to each non-overlapping period is used as the new instantaneous resource spike data to complete the update of instantaneous resource spike data;

[0114] S6: Update the initial monitoring frequency based on instantaneous resource spike data;

[0115] In S6, the process of updating the initial monitoring frequency based on instantaneous resource spike data is as follows:

[0116] The instantaneous resource spike data is re-marked by time period, and the shortest instantaneous resource spike time period is extracted. The time length corresponding to the shortest instantaneous resource spike time period is used as the new monitoring frequency to complete the update of the initial monitoring frequency.

[0117] S7: Monitor the instantaneous resource spikes and traffic data of the computer system according to the updated initial monitoring frequency, and predict the instantaneous resource spikes of the computer system in future time periods by combining the LSTM model.

[0118] In S7, the process of predicting instantaneous resource spikes in a computer system over a future period is as follows:

[0119] S71, real-time monitoring of instantaneous resource spike data and flow data of computer system according to the updated initial monitoring frequency, and conversion of instantaneous resource spike data and flow data into LSTM input format sequence samples after normalization processing, wherein the sequence samples include spike duration sequence samples and flow data samples.

[0120] S72, set the historical window k (e.g., the past 10 minutes) as the input feature to capture contextual dependencies; and the prediction window d (e.g., the next 1 minute) as the output target;

[0121] S73, slide a window over the sequence samples in chronological order to cut the continuous instantaneous resource spike data and flow data into input feature samples and output feature samples. The input feature samples include the spike duration sequence and flow sequence within the historical window k, and the output feature samples include the spike duration sequence within the prediction window d.

[0122] For example: if the historical window k is 10 time steps (1 minute per time step), then each input feature sample contains the duration of the spikes and the flow data for the previous 10 minutes, and the output feature sample includes the duration of the spikes for 1 minute.

[0123] S74, construct an LSTM model, including an input layer, an LSTM layer, and a fully connected layer;

[0124] The input layer is used to receive input feature samples and output feature samples;

[0125] An LSTM layer contains 1-2 stacked LSTM units, used to extract long-term dependencies between input and output feature samples.

[0126] The fully connected layer maps the LSTM output to the prediction dimension through the Dense layer;

[0127] A dataset is constructed by inputting feature samples and outputting feature samples. The dataset is then divided into training and testing sets in a 7:3 ratio. The LSTM model is trained and tested using the training and testing sets to obtain a trained LSTM model.

[0128] S75 acquires the latest glitch duration and flow data in real time, updates the input feature samples, and inputs them into the LSTM model to predict the glitch duration in future periods.

[0129] For example, the duration and flow data of the glitch corresponding to the 10 latest time steps are obtained, the input feature samples are updated, and the duration of the glitch in the next minute is predicted.

[0130] The technical solution of this invention is as follows: By performing time-period marking processing on the instantaneous resource spike data obtained from monitoring the computer system during its running period according to the initial monitoring frequency, and by identifying and marking the traffic data monitored in real-time during the computer system's running period, an instantaneous resource spike time period library and an abnormal traffic inflow time period library are obtained. Overlap analysis and correlation analysis are then performed to determine whether the instantaneous resource spike phenomenon in the computer system is caused by abnormal traffic inflow. If so, the overlapping and non-overlapping time periods obtained from the overlap analysis of the instantaneous resource spike time period library and the abnormal traffic inflow time period library are compared to determine whether... This invention addresses the need to update the instantaneous resource spike data of a computer system during its runtime. Based on this data, the initial monitoring frequency is updated. The updated initial monitoring frequency is then used to monitor the instantaneous resource spike data and traffic data of the computer system. Finally, an LSTM model is used to predict instantaneous resource spikes in future time periods. The invention primarily addresses updating the initial monitoring frequency of the computer system to prevent it from failing to accurately capture or monitor instantaneous resource spikes. After updating the initial monitoring frequency, the instantaneous resource spike data and traffic data monitored according to the updated initial monitoring frequency are combined with an LSTM model to predict instantaneous resource spikes in future time periods, thereby improving the accuracy of instantaneous resource spike prediction.

[0131] Example 2: Please refer to Figure 2 As shown in the embodiment of the present invention, an intelligent prediction system based on big data analysis includes the following modules:

[0132] Instantaneous resource glitch marking module: By marking the instantaneous resource glitch data obtained from monitoring the computer system during its running period according to the initial monitoring frequency, a database of instantaneous resource glitch time periods is obtained;

[0133] Traffic anomaly inflow marking module: performs anomaly identification and time period marking on the traffic data of the computer system during the running period, and obtains a traffic anomaly inflow time period library;

[0134] The correlation analysis module performs overlap and correlation analysis on the instantaneous resource spike time period database and the abnormal traffic inflow time period database to determine whether the instantaneous resource spike phenomenon in the computer system is caused by abnormal traffic inflow.

[0135] Data update judgment module: If yes, compare the overlapping and non-overlapping time periods obtained by the overlap analysis of the instantaneous resource spike time period library and the abnormal traffic inflow time period library to determine whether the instantaneous resource spike data needs to be updated. If it needs to be updated, enter the spike data update module; if it does not need to be updated, enter the monitoring and adjustment update module.

[0136] The glitch data update module updates the instantaneous resource glitch data and then enters the monitoring, adjustment and update module after the update.

[0137] Monitoring, adjustment and update module: Updates the initial monitoring frequency based on instantaneous resource spike data;

[0138] Instantaneous resource spike prediction module: Monitors instantaneous resource spike data and traffic data of the computer system according to the updated initial monitoring frequency, and predicts instantaneous resource spikes of the computer system in future time periods by combining the LSTM model.

[0139] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. An intelligent prediction method based on big data analysis, characterized in that: include: S1: By performing time period marking processing on the instantaneous resource spike data obtained by monitoring the computer system during its running period according to the initial monitoring frequency, an instantaneous resource spike time period library is obtained; The instantaneous resource spike data represents the duration of the instantaneous resource spike phenomenon caused by the computer system's resource request volume exceeding the preset standard request volume. S2: Perform anomaly identification and time period marking on the traffic data of the computer system during its running period to obtain a database of abnormal traffic inflow time periods; S3: Perform overlap and correlation analysis on the instantaneous resource spike time period database and the abnormal traffic inflow time period database to determine whether the instantaneous resource spike phenomenon in the computer system is caused by abnormal traffic inflow. S4: If so, compare the overlapping and non-overlapping periods obtained by the overlap analysis of the instantaneous resource spike period database and the abnormal traffic inflow period database to determine whether the instantaneous resource spike data needs to be updated. If so, proceed to S5; otherwise, proceed to S6. S5: Update the instantaneous resource spike data, and proceed to S6 after the update; S6: Update the initial monitoring frequency based on instantaneous resource spike data; S7: Monitor the instantaneous resource spikes and traffic data of the computer system according to the updated initial monitoring frequency, and predict the instantaneous resource spikes of the computer system in future time periods by combining the LSTM model.

2. The intelligent prediction method based on big data analysis according to claim 1, characterized in that: The instantaneous resource spike time period library is obtained as follows: Based on the start and end times of each instance of instantaneous resource spikes, the time periods of instantaneous resource spikes are determined and compiled into a database of instantaneous resource spike time periods.

3. The intelligent prediction method based on big data analysis according to claim 1, characterized in that: The traffic data represents the amount of data transmitted by the computer system in real time per unit time. The method for obtaining the database of abnormal traffic inflow periods is as follows: Obtain the data transmission volume at different unit time points during the normal operation period of historical traffic, and calculate the mean and standard deviation to obtain the mean and standard deviation of traffic during the normal operation period of historical traffic. use The principle is to calculate the traffic anomaly score based on the real-time data transmission volume of the computer system at different unit times during the running period, the average traffic volume during the historical normal operation period, and the standard deviation of traffic volume. If the traffic anomaly score does not meet the requirements, it indicates a traffic anomaly. The traffic anomaly influx period is determined based on the start and end times of the traffic anomaly, and the traffic anomaly influx period database is compiled.

4. The intelligent prediction method based on big data analysis according to claim 1, characterized in that: The process of the overlap analysis is as follows: If any abnormal traffic inflow period in the abnormal traffic inflow period database is the same as the period corresponding to the instantaneous resource spike period, then the instantaneous resource spike period is marked as an overlapping period; otherwise, it is marked as a non-overlapping period.

5. The intelligent prediction method based on big data analysis according to claim 1, characterized in that: The process of correlation analysis is as follows: Obtain the resource request volume and data transmission volume within different units of time during overlapping periods, and integrate them according to time sequence to obtain the resource request volume sequence and data transmission volume sequence; The Pearson correlation coefficient method was used to calculate the resource request sequence and the data transmission sequence; If the Pearson correlation coefficient is greater than or equal to the Pearson correlation coefficient threshold, the overlapping period is marked as the period of influence.

6. The intelligent prediction method based on big data analysis according to claim 5, characterized in that: The process of determining whether the transient resource spikes in the computer system are caused by abnormal traffic inflows is as follows: The proportion of overlapping periods within the instantaneous resource spike period is used to obtain the first impact value; The proportion of the number of periods with statistical impact within the overlapping periods is used to obtain the second impact value; The sum of the first and second impact values ​​is used to obtain the impact value of the inflow. If the impact value of the traffic influx meets the requirements, it means that the instantaneous resource spikes in the computer system are caused by abnormal traffic influx.

7. The intelligent prediction method based on big data analysis according to claim 1, characterized in that: The process of comparing overlapping and non-overlapping time periods is as follows: Among all overlapping time periods, the overlapping time period with the shortest duration is selected as the reference time period; If the duration of any non-overlapping time period is shorter than the duration of the reference time period, it indicates that the instantaneous resource spike data of the computer system during the running period needs to be updated. If the duration of all non-overlapping time periods is longer than the duration of the reference time period, it means that there is no need to update the instantaneous resource spike data of the computer system during the running period.

8. The intelligent prediction method based on big data analysis according to claim 7, characterized in that: The process of updating the instantaneous resource spike data is as follows: The time length corresponding to each non-overlapping period is used as the new instantaneous resource spike data to complete the update of instantaneous resource spike data; The process of updating the initial monitoring frequency is as follows: Extract the time length corresponding to the shortest instantaneous resource spike period as the new monitoring frequency to complete the update of the initial monitoring frequency.

9. The intelligent prediction method based on big data analysis according to claim 1, characterized in that: The process of predicting instantaneous resource spikes in the computer system over future time periods is as follows: S71 performs real-time monitoring of instantaneous resource spike data and traffic data of the computer system according to the updated initial monitoring frequency, and converts them into sequence samples in LSTM input format, wherein the sequence samples include spike duration sequence samples and traffic data samples; S72, set the historical window k as the input feature and the prediction window d as the output target; S73, slide a window over the sequence samples in chronological order to cut the continuous instantaneous resource spike data and flow data into input feature samples and output feature samples. The input feature samples include the spike duration sequence and flow sequence within the historical window k, and the output feature samples include the spike duration sequence within the prediction window d. S74, construct an LSTM model, including an input layer, an LSTM layer, and a fully connected layer; The input layer is used to receive input feature samples and output feature samples; An LSTM layer contains 1-2 stacked LSTM cells; The fully connected layer maps the LSTM output to the prediction dimension through the Dense layer; The LSTM model is trained and tested by input feature samples and output feature samples to obtain a trained LSTM model. S75 acquires the latest glitch duration and flow data in real time, updates the input feature samples, and inputs them into the LSTM model to predict the glitch duration in future periods.

10. An intelligent prediction system based on big data analysis, characterized in that: The system is used to execute the intelligent prediction method according to any one of claims 1-9, comprising: Instantaneous resource glitch marking module: By marking the instantaneous resource glitch data obtained from monitoring the computer system during its running period according to the initial monitoring frequency, a database of instantaneous resource glitch time periods is obtained; Traffic anomaly inflow marking module: performs anomaly identification and time period marking on the traffic data of the computer system during the running period, and obtains a traffic anomaly inflow time period library; The correlation analysis module performs overlap and correlation analysis on the instantaneous resource spike time period database and the abnormal traffic inflow time period database to determine whether the instantaneous resource spike phenomenon in the computer system is caused by abnormal traffic inflow. Data update judgment module: If yes, compare the overlapping and non-overlapping time periods obtained by the overlap analysis of the instantaneous resource spike time period library and the abnormal traffic inflow time period library to determine whether the instantaneous resource spike data needs to be updated. If it needs to be updated, enter the spike data update module; if it does not need to be updated, enter the monitoring and adjustment update module. The glitch data update module updates the instantaneous resource glitch data and then enters the monitoring, adjustment and update module after the update. Monitoring, adjustment and update module: Updates the initial monitoring frequency based on instantaneous resource spike data; Instantaneous resource spike prediction module: Monitors instantaneous resource spike data and traffic data of the computer system according to the updated initial monitoring frequency, and predicts instantaneous resource spikes of the computer system in future time periods by combining the LSTM model.

Images