Unified operation and maintenance method based on big data

Abstract

The invention relates to the technical field of big data processing and analysis, in particular to a unified operation and maintenance method based on big data. The method comprises the following steps: S1, automatically collecting various operation and maintenance data from an IT system through a log management module and a performance monitoring module; s2, performing data preprocessing on the collected operation and maintenance data to remove invalid or abnormal data; s3, performing data analysis on the operation and maintenance data after data preprocessing to find potential problems and trends; and S4, making an intelligent decision according to a data analysis result, and executing an automatic response. And S5, displaying a data analysis result in a chart form. Potential problems and trends can be identified through data analysis, and possible problems in the future can be predicted. Automatic response is executed, manual intervention time is shortened, measures can be taken rapidly, and the influence of faults on services is reduced.

Description

Technical Field

[0001] This invention relates to the field of big data processing and analysis technology, and more specifically, to a unified operation and maintenance method based on big data. Background Technology

[0002] A unified operations and maintenance (O&M) approach based on big data is a method that comprehensively utilizes big data technology and O&M management concepts to improve the efficiency and quality of IT system operations and maintenance. This approach aims to achieve unified management and optimization of IT system O&M work through automation and intelligent methods, thereby increasing O&M efficiency, reducing costs, and improving service quality.

[0003] Existing unified operation and maintenance methods based on big data suffer from latency when processing large amounts of data, inability to monitor system status in real time, failure to promptly identify problems, and the need for manual intervention for maintenance. Furthermore, some potential problems and trends cannot be identified in a timely manner, and the lack of reporting and visualization makes it difficult for operation and maintenance personnel to understand complex data issues. Therefore, a unified operation and maintenance method based on big data is needed. Summary of the Invention

[0004] The purpose of this invention is to provide a unified operation and maintenance method based on big data, in order to solve the problems mentioned in the background art, such as the delay when processing large amounts of data, the inability to monitor the system status in real time and detect problems in a timely manner, the need for manual intervention for maintenance, the inability to identify some potential problems and trends in a timely manner, and the lack of reports and explanations, which makes it difficult for operation and maintenance personnel to understand complex data.

[0005] To achieve the above objectives, the present invention aims to provide a unified operation and maintenance method based on big data, comprising the following steps:

[0006] S1. Automatically collect various operational data from the IT system through the log management module and performance monitoring module;

[0007] Among them, various operational data include server and application log files, as well as server CPU utilization and memory usage.

[0008] S2. Perform data preprocessing on the collected operation and maintenance data to remove invalid or abnormal data and integrate the data;

[0009] S3. Perform data analysis on the preprocessed operation and maintenance data to identify potential problems and trends;

[0010] S4. Make intelligent decisions based on data analysis results and execute automated responses;

[0011] S5. Display the data analysis results in the form of charts and generate a report.

[0012] As a further improvement to this technical solution, in S1, the log management module and the performance monitoring module are specifically as follows:

[0013] The log management module is used to collect log files from the server and applications.

[0014] The performance monitoring module is used to monitor server CPU utilization and memory usage.

[0015] As a further improvement to this technical solution, the specific steps of data preprocessing in step S2 are as follows:

[0016] S2.1 Check if there are duplicate items in the dataset and remove duplicate data entries;

[0017] S2.2 Detect whether there are missing values ​​in the dataset and fill in the missing values ​​using the mean method;

[0018] S2.3 Use IQR statistical methods to detect outliers and correct them;

[0019] S2.4 Convert non-numerical data into numerical data using label encoding, and then scale the numerical data to the same scale using the minimum-maximum scaling method;

[0020] S2.5. Convert the data into a standard normal distribution with a mean of 0 and a variance of 1.

[0021] S2.6 Finally, integrate all kinds of operation and maintenance data into the same dataset.

[0022] As a further improvement to this technical solution, the IQR statistical method in S2.3 is specifically as follows:

[0023] Outliers={x|x<Q1-1.5×IQR orx>Q3+1.5×IQR};

[0024] Where Outliers represents the set of detected outliers; x represents each data point in the dataset; Q1 represents the first quartile; Q3 represents the third quartile; and IQR represents the interquartile range.

[0025] As a further improvement to this technical solution, the specific steps for data analysis of the preprocessed data in step S3 are as follows:

[0026] S3.1 Use the ARIMA model to identify the changing trends of operation and maintenance data over time;

[0027] S3.2. Then use moving average and exponential smoothing methods to smooth the data to reveal potential trends;

[0028] S3.3 Based on the outliers detected in S2.3, the Z-score method is used to further confirm the outliers;

[0029] S3.4 Calculate the correlation coefficients between different variables to identify which variables are associated. As a further improvement to this technical solution, in S3.1, the ARIMA model is specifically as follows:

[0030] S t =φ1S t-1 +φ2S t-2 +…+φ p S t-p +∈ t +θ1∈ t-1 +…+θ q ∈ t-q ;

[0031] Among them, S t φ represents the value of the time series at time t; i θ represents the coefficient of the autoregressive term. i represents the coefficient of the moving average term; p represents the number of autoregressive terms; q represents the number of moving average terms; ∈ t This represents the value of the white noise process at time t.

[0032] As a further improvement to this technical solution, in S3.2, the data smoothing method using moving average and exponential smoothing is specifically as follows:

[0033] Moving average:

[0034]

[0035] Among them, SMA t X represents the moving average at time t; m represents the window size; X i This represents the original data value at time i;

[0036] Exponential smoothing:

[0037] ES t =αX t +(1-α)ES t-1 ;

[0038] Among them, ES t X represents the exponentially smoothed value at time t; α represents the smoothing factor; X t This represents the original data value at time t.

[0039] As a further improvement to this technical solution, the Z-score method in S3.3 is specifically as follows:

[0040] Outliers = {x||z(x)|>z} threshold};

[0041]

[0042] Where z(x) represents the standardized score of data point x; μ represents the mean of the dataset; σ represents the standard deviation of the dataset; Z threshold This represents the threshold.

[0043] As a further improvement to this technical solution, the specific steps for making intelligent decisions and executing automated responses in step S4 are as follows:

[0044] S4.1 Set thresholds and trigger conditions to automatically trigger a response mechanism when a specific event occurs;

[0045] S4.2. Dynamically adjust operation and maintenance strategies based on the results of real-time analysis;

[0046] S4.3 When a problem is detected, attempt to automatically repair it to reduce the need for manual intervention.

[0047] As a further improvement to this technical solution, in step S5, the specific steps for displaying the data in the form of charts and generating a report are as follows:

[0048] S5.1 Use dashboards and charts to display operational data and trends;

[0049] S5.2. Regularly generate analysis reports;

[0050] The analysis report includes an overview of operation and maintenance data, anomaly detection results, and trend analysis.

[0051] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0052] 1. This unified operation and maintenance method based on big data automatically collects various operation and maintenance data from the IT system through log management and performance monitoring modules. This ensures the acquisition of all important operation and maintenance data and enables real-time monitoring of system status to promptly identify problems. Data preprocessing removes invalid or abnormal data, improving the accuracy of subsequent analysis.

[0053] 2. This unified operations and maintenance method based on big data can identify potential problems and trends through data analysis, helping to predict future issues. It executes automated responses, reducing manual intervention time and enabling rapid action to mitigate the impact of failures on business operations. Results are presented in graphical form, making complex data easier for operations personnel to understand. The generated reports can serve as historical records for future reference and auditing. Attached Figure Description

[0054] Figure 1 This is a flowchart of the overall method of the present invention. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] Example:

[0057] Please see Figure 1 As shown, this embodiment provides a unified operation and maintenance method based on big data, including the following steps:

[0058] S1. Automatically collect various operational data from the IT system through the log management module and performance monitoring module;

[0059] Among them, various operational data include server and application log files, as well as server CPU utilization and memory usage.

[0060] In this example, the log management module and the performance monitoring module are specifically as follows:

[0061] The log management module is used to collect log files from the server and applications.

[0062] The performance monitoring module is used to monitor server CPU utilization and memory usage.

[0063] S2. Perform data preprocessing on the collected operation and maintenance data to remove invalid or abnormal data and integrate the data;

[0064] In this example, the specific steps of data preprocessing are as follows:

[0065] S2.1 Check if there are duplicate items in the dataset and remove duplicate data entries;

[0066] The specific steps are as follows:

[0067] import pandas as pd

[0068] #Load data

[0069] data=pd.read_csv(′data.csv′)

[0070] #Detect and remove duplicates

[0071] data_unique=data.drop_duplicates()

[0072] #View results

[0073] print(data_unique.head())

[0074] S2.2 Detect whether there are missing values ​​in the dataset and fill in the missing values ​​using the mean method;

[0075] The specific steps are as follows:

[0076] import pandas as pd

[0077] #Load data

[0078] data=pd.read_csv(′data.Csv′)

[0079] #Detect and fill missing values

[0080] data_filled=data.fillna(data.mean())

[0081] #View results

[0082] print(data_filled.head())

[0083] S2.3 Use IQR statistical methods to detect outliers and correct them;

[0084] In this example, the IQR statistical method is as follows:

[0085] Outliers={x|x<Q1-1.5×IQR or x>Q3+1.5×IQR};

[0086] Where Outliers represents the set of detected outliers; x represents each data point in the dataset; Q1 represents the first quartile; Q3 represents the third quartile; and IQR represents the interquartile range.

[0087] Specifically, IQR (Interquartile Range) is a statistic that describes the dispersion of data; it represents the range of the middle 50% of a set of data. In unified operations and maintenance methods based on big data, IQR can help identify outliers and monitor changes in system performance metrics.

[0088] S2.4 Convert non-numerical data into numerical data using label encoding, and then scale the numerical data to the same scale using the minimum-maximum scaling method;

[0089] Among them, the minimum-maximum scaling method is used to scale numerical features. It uses the maximum and minimum values ​​in the dataset to linearly transform each feature value, ensuring that all scaled feature values ​​are within the specified range, thereby making features of different orders of magnitude or different units comparable.

[0090] S2.5. Convert the data into a standard normal distribution with a mean of 0 and a variance of 1.

[0091] S2.6 Finally, integrate all kinds of operation and maintenance data into the same dataset.

[0092] S3. Perform data analysis on the preprocessed operation and maintenance data to identify potential problems and trends;

[0093] In this example, the specific steps for data analysis of the preprocessed data are as follows:

[0094] S3.1 Use the ARIMA model to identify the changing trends of operation and maintenance data over time;

[0095] In this example, the ARIMA model is specifically as follows:

[0096] S t =φ1S t-1 +φ2S t-2 +…+φ p S t-p +∈ t +θ1∈ t-1 +…+θ q ∈ t-q ;

[0097] Among them, S t φ represents the value of the time series at time t; i θ represents the coefficient of the autoregressive term. i represents the coefficient of the moving average term; p represents the number of autoregressive terms; q represents the number of moving average terms; ∈ t This represents the value of the white noise process at time t.

[0098] S3.2. Then use moving average and exponential smoothing methods to smooth the data to reveal potential trends;

[0099] In this example, the data is smoothed using moving average and exponential smoothing methods as follows:

[0100] Moving average:

[0101]

[0102] Among them, SMA t X represents the moving average at time t; m represents the window size; Xi This represents the original data value at time i;

[0103] Exponential smoothing:

[0104] ES t =αX t +(1-α)ES t-1 ;

[0105] Among them, ES t X represents the exponentially smoothed value at time t; α represents the smoothing factor; X t This represents the original data value at time t.

[0106] S3.3 Based on the outliers detected in S2.3, the Z-score method is used to further confirm the outliers;

[0107] In this example, the Z-score method is specifically as follows:

[0108] Outliers = {x||z(x)|>z} threshold};

[0109]

[0110] Where z(x) represents the standardized score of data point x; μ represents the mean of the dataset; σ represents the standard deviation of the dataset; z threshold This represents the threshold.

[0111] S3.4 Calculate the correlation coefficients between different variables to identify which variables are associated.

[0112] The correlation coefficient is specifically:

[0113]

[0114] Where, x i and y i These are the observed values ​​of the variables; and These are the means of the variables.

[0115] S4. Make intelligent decisions based on data analysis results and execute automated responses;

[0116] In this example, the specific steps for making intelligent decisions and executing automated responses are as follows:

[0117] S4.1 Set thresholds and trigger conditions to automatically trigger a response mechanism when a specific event occurs;

[0118] S4.2. Dynamically adjust operation and maintenance strategies based on the results of real-time analysis;

[0119] S4.3 When a problem is detected, attempt to automatically repair it to reduce the need for manual intervention.

[0120] S5. Display the data analysis results in the form of charts and generate a report.

[0121] In this example, the specific steps to display the data in chart form and generate a report are as follows:

[0122] S5.1 Use dashboards and charts to display operational data and trends;

[0123] S5.2. Regularly generate analysis reports;

[0124] The analysis report includes an overview of operation and maintenance data, anomaly detection results, and trend analysis.

[0125] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.

Claims

1. A unified operation and maintenance method based on big data, characterized in that: Includes the following steps: S1. Automatically collect various operation and maintenance data from the IT system through the log management module and performance monitoring module; Among them, various operational data include server and application log files, as well as server CPU utilization and memory usage. S2. Perform data preprocessing on the collected operation and maintenance data to remove invalid or abnormal data and integrate the data; S3. Perform data analysis on the preprocessed operation and maintenance data to identify potential problems and trends; S4. Make intelligent decisions based on data analysis results and execute automated responses; S5. Display the data analysis results in the form of charts and generate a report.

2. The unified operation and maintenance method based on big data according to claim 1, characterized in that: In step S1, the log management module is used to collect log files from the server and applications. The performance monitoring module is used to monitor server CPU utilization and memory usage.

3. The unified operation and maintenance method based on big data according to claim 1, characterized in that: In step S2, the specific steps of data preprocessing are as follows: S2.1 Check if there are duplicate items in the dataset and remove duplicate data entries; S2.2 Detect whether there are missing values ​​in the dataset and fill in the missing values ​​using the mean method; S2.3 Use IQR statistical methods to detect outliers and correct them; S2.4 Convert non-numerical data into numerical data using label encoding, and then scale the numerical data to the same scale using the minimum-maximum scaling method; S2.

5. Convert the data into a standard normal distribution with a mean of 0 and a variance of 1. S2.6 Finally, integrate all kinds of operation and maintenance data into the same dataset.

4. The unified operation and maintenance method based on big data according to claim 3, characterized in that: In S2.3, the IQR statistical method is as follows: Outliers={x|x<Q1-1.5×IQR or x>Q3+1.5×IQR}; Where Outliers represents the set of detected outliers; x represents each data point in the dataset; Q1 represents the first quartile; Q3 represents the third quartile; and IQR represents the interquartile range.

5. The unified operation and maintenance method based on big data according to claim 1, characterized in that: In step S3, the specific steps for data analysis of the preprocessed data are as follows: S3.1 Use the ARlMA model to identify the changing trends of operation and maintenance data over time; S3.

2. Then use moving average and exponential smoothing methods to smooth the data; S3.3 Based on the outliers detected in S2.3, the Z-score method is used to further confirm the outliers; S3.4 Calculate the correlation coefficients between different variables to identify which variables are associated.

6. The unified operation and maintenance method based on big data according to claim 1, characterized in that: In S3.1, the ARlMA model is specifically as follows: S t =φ1S t-1 +φ2S t-2 +…+φ p S t-p +∈ t +θ1∈ t-1 +…+θ q ∈ t-q ; Among them, S t φ represents the value of the time series at time t; i The coefficient of the autoregressive term; (The last part is incomplete and likely refers to a separate topic.) i denoted by ; p represents the number of autoregressive terms; q represents the number of moving average terms; ∈t represents the value of the white noise process at time t.

7. The unified operation and maintenance method based on big data according to claim 1, characterized in that: In S3.2, the data smoothing method using moving average and exponential smoothing is specifically as follows: Moving average: Among them, SMA t X represents the moving average at time t; m represents the window size; X i This represents the original data value at time i; Exponential smoothing: IS t =αX t +(1-α)ES t-1 ; Among them, ES t X represents the exponentially smoothed value at time t; α represents the smoothing factor; X t This represents the original data value at time t.

8. The unified operation and maintenance method based on big data according to claim 1, characterized in that: In S3.3, the Z-score method is specifically as follows: Outliers={x||z(x)|>z threshold }; Where z(x) represents the standardized score of data point x; μ represents the mean of the dataset; σ represents the standard deviation of the dataset; z threshold This represents the threshold.

9. The unified operation and maintenance method based on big data according to claim 1, characterized in that: In step S4, the specific steps for making intelligent decisions and executing automated responses are as follows: S4.1 Set thresholds and trigger conditions to automatically trigger a response mechanism when a specific event occurs; S4.

2. Dynamically adjust operation and maintenance strategies based on the results of real-time analysis; S4.3 When a problem is detected, attempt to automatically repair it to reduce the need for manual intervention.

10. The unified operation and maintenance method based on big data according to claim 1, characterized in that: In step S5, the specific steps for displaying the data in chart form and generating a report are as follows: S5.1 Use dashboards and charts to display operational data and trends; S5.

2. Regularly generate analysis reports; The analysis report includes an overview of operation and maintenance data, anomaly detection results, and trend analysis.

Images