Cloud platform cluster network security log data fault prediction processing method

Abstract

The invention discloses a cloud platform cluster network security log data fault prediction processing method, which comprises the following steps of: firstly, acquiring original log data of a cloud data computing system, and then performing normalization processing on the original log data; further performing line analysis and identification on the normalized original log data, and screening out abnormal log data; and finally, further analyzing the abnormal log data, particularly identifying abnormal stack information and determining a fault source. By processing and analyzing the log data of the cloud data computing system, the system fault is efficiently predicted and identified, so that the operation safety of the system is improved; by monitoring and analyzing the log data in real time, measures can be taken in time to repair or optimize before the fault occurs, the influence of the fault on system services and users is reduced, and the availability and stability of the system are improved.

Description

Technical Field

[0001] The present invention relates to the field of cloud computing, and in particular to a method for predicting and processing network security log data faults in a cloud platform cluster. Background Art

[0002] Cloud data computing systems are widely used. For example, they can be used in e-commerce backend order matching systems and e-commerce platform statistics systems (such as distributed cluster systems).

[0003] Generally speaking, server systems (e.g., cloud computing systems) are prone to system failures during large-scale operations. These failures can be caused by a variety of factors, including network system crashes (network failures), system errors, application server cluster node crashes, and configuration errors. They can also be caused by system security issues, such as intrusions, hacker attacks, and virus infections.

[0004] Therefore, it is crucial to inspect text logs. A common approach is to monitor log files: set up monitoring programs to regularly check the status and content of log files to ensure that they are recording properly. Therefore, comprehensive inspection and monitoring of the log system is necessary to promptly identify and resolve issues and ensure the normal operation of the system.

[0005] There are currently the following ways to monitor log files: Log data reflects the operating status of the system (server system or software application system). Analyzing the system's operating log data, extracting fault characteristics, and performing fault identification and processing are common methods for system fault diagnosis;

[0006] In particular, system disaster failures caused by network system crashes (network failures), system errors, application server cluster node crashes, configuration errors, etc. are the most frequent. When the server is at a certain time stage (peak computing period), it may be easy to have log failures caused by system errors, application server cluster node crashes, and configuration errors.

[0007] Further research found that various operating conditions may occur during system operation (for example, application server cluster node crashes, configuration errors), and the logs generated are not the same. In this case, it is crucial to extract fault features from complex and changeable log data and separate and identify fault logs. However, research found that the processing methods in existing technologies are mostly centered on log analysis of conventional network faults. The constrained identification method of conventional network fault identification cannot fully identify whether there are anomalies in the log data, which brings great pressure to log fault identification. Summary of the Invention

[0008] The present invention proposes a cloud platform cluster network security log data fault prediction and processing method, which includes the following steps:

[0009] Get raw log data;

[0010] Normalizing the stack nodes executed by the application server cluster nodes and the corresponding network fault log data corresponding to the original log data to obtain normalized original log data;

[0011] Analyze and identify the normalized original log data and filter out abnormal log data;

[0012] Identify the exception stack information in the exception log data to determine the source of the fault.

[0013] Preferably, the normalized original log data includes a normalized stack node original log data set and a normalized network fault log data set.

[0014] Preferably, normalizing the stack nodes executed by the application server cluster nodes corresponding to the original log data and the corresponding network fault log data to obtain the normalized original log data includes the following steps:

[0015] Normalization processing is performed on the stack nodes executed by the application server cluster nodes corresponding to the original log data to obtain normalized log data; and normalization processing is performed on the network fault log data corresponding to the original log data to obtain normalized log data.

[0016] Preferably, the analyzing and identifying the normalized original log data to obtain abnormal log data by screening includes the following steps:

[0017] During the initialization process, the normalized original log data is first analyzed and identified to obtain a first classification result; abnormal log data is screened from the first classification result;

[0018] Verifying the first classification result to obtain a verification result; determining whether the verification result meets the target expansion condition, and if so, initiating a third-level weight vector introduction operation;

[0019] The normalized original log data is analyzed and identified for the second time in combination with the third-level weight vector to obtain a second classification result; and abnormal log data is screened from the second classification result.

[0020] Preferably, during the initialization process, the normalized original log data is analyzed and identified for the first time to obtain a first classification result, which includes the following steps:

[0021] Obtain historical data, select m typical fault log samples from the historical data to construct a typical fault log sample set M = {m1, m2, m3, ...mi}; where mi represents the i-th typical fault log sample; and obtain j feature vectors corresponding to each of the typical fault log samples to construct a feature vector set K = {k1, k2, k3, ...kj}; where kj is the j-th feature vector;

[0022] Construct a target matrix A based on all the typical fault log sample sets and all the feature vector sets K:

[0023] Among them, mikj is the jth eigenvector of the i-th typical fault sample log;

[0024] Setting a network fault weight vector W = {w1, w2, wj..., wt} for each of the feature vectors; wherein wj is the weight of the j-th feature vector;

[0025] For each of the feature vectors, a stack fault weight vector N = {n1, n2, ..., nj, ng} is constructed; wherein nj is the weight of the j-th feature in the exception stack;

[0026] Obtaining a fault feature vector F by calculation based on the network fault weight vector W and the stack fault weight vector N;

[0027] The fault feature vector F is calculated as follows:

[0028] F=W+N;

[0029] Among them, W+N is the addition of corresponding elements;

[0030] Calculate and obtain a comprehensive fault classification vector Y corresponding to each typical fault log sample based on the target matrix A and the fault feature vector F; input the comprehensive fault classification vector Y into a pre-trained classifier to output a first classification result;

[0031] The calculation method of the comprehensive fault classification vector Y is:

[0032] Y = F × A = {y1, y2, y3, ...yi, ...ym}; where yi is the comprehensive fault classification vector of the i-th typical fault log sample;

[0033] The first classification result is used to indicate whether the current i-th sample has a risk;

[0034] Preferably, the determining whether the verification result meets the target expansion condition, and if so, starting the third-level weight vector introduction operation, includes the following steps:

[0035] Pre-setting a secondary evaluation threshold; the secondary evaluation threshold includes an error rate standard threshold and a stack fault evaluation standard threshold;

[0036] Verifying the first classification result of the cloud data computing system using the secondary evaluation threshold to obtain a verification result;

[0037] Determine whether the verification result meets the preset verification error rate condition, and if so, start the third-level weight vector introduction operation;

[0038] The verification error rate condition refers to: the overall classification error rate is higher than the error rate standard threshold and the stack fault judgment error rate is higher than the stack fault evaluation standard threshold;

[0039] Preferably, the verifying the first classification result of the cloud data computing system by using the secondary evaluation threshold to obtain the verification result includes the following steps:

[0040] Calculating an overall classification error rate based on the first classification result; determining whether the overall classification error rate is higher than an error rate standard threshold; and if so, further screening multiple abnormal log data indicating actual stack failures;

[0041] Count the number of abnormal log data with actual stack faults, calculate the ratio of the number of samples of the first classification results of the abnormal log data with predicted errors to the abnormal log data with actual stack faults; and determine whether the stack fault judgment error rate is greater than the stack fault evaluation standard threshold.

[0042] Preferably, the second analysis and identification of the normalized original log data in combination with the third-level weight vector to obtain a second classification result specifically includes:

[0043] Obtain e extended feature vectors K'={k'1, k'2, k'3, ..., k'e} corresponding to the typical fault log sample; wherein k'e is the e-th extended feature vector; construct an extended target matrix A' based on the extended feature K' and the typical fault log sample set M;

[0044] The expanded target matrix A' is expressed as:

[0045] Among them, mik'e is the e-th extended feature vector of the i-th typical fault sample log;

[0046] Presetting an extended network fault weight vector W'={w'1, w'2, w'j..., w't} for each extended feature vector; wherein w'j is the weight of the j-th extended feature vector in the network fault;

[0047] Constructing an extended stack fault weight vector N'={n'1, n'2, ... n'j, n'g} for each extended feature vector; wherein n'j is the weight of the j-th extended feature vector in the exception stack;

[0048] Constructing a third-level fault weight vector P = {p1, p2, .. pj, pf} for each of the extended feature vectors; wherein pf is the weight of the f-th extended feature vector in the third-level fault;

[0049] An extended fault feature vector F' is calculated based on the extended network fault weight vector W', the extended stack fault weight vector N', and the third-level fault weight vector P (or the weight of the access weight vector mentioned above);

[0050] The calculation method of the extended fault feature vector F' is:

[0051] F'=W'+N'+P;

[0052] Among them, W'+N'+P is the addition of corresponding elements;

[0053] Calculating and obtaining an expanded comprehensive fault classification vector Y' corresponding to each of the typical fault log samples based on the expanded target matrix A' and the expanded fault vector F'; inputting the expanded comprehensive fault classification vector Y' into a pre-trained classifier to output a second classification result;

[0054] The calculation method of the extended comprehensive fault classification vector Y' is:

[0055] Y'=F'×A'={y'1, y'2, y'3, ...y'i, ...y'm}; where y'i is the comprehensive fault classification vector of the i-th typical fault log sample;

[0056] The second classification result is used to indicate whether the current i-th sample has a risk.

[0057] Preferably, the step of constructing the third-level fault weight vector P={p1, p2, .. pj, pf} for each of the extended feature vectors includes the following steps:

[0058] Obtaining the third-level correlation features corresponding to each of the typical fault log samples;

[0059] Normalizing the third-level correlation features to obtain normalized correlation feature data;

[0060] Performing statistics on the normalized correlation feature data to obtain distribution information corresponding to the third-level correlation feature;

[0061] A weight is assigned to each of the typical fault log samples based on the distribution information to obtain a third-level fault weight vector corresponding to each of the extended feature vectors.

[0062] Compared with the prior art, the embodiments of the present invention have at least the following technical advantages:

[0063] Analysis of the cloud platform cluster network security log data fault prediction and processing method provided by the present invention shows that in specific applications:

[0064] First, we obtain the original log data of the cloud data computing system. This data contains various records and information during the system's operation. Then, we normalize the original log data to convert different formats and types of log data into the same data format and unit for subsequent analysis and processing.

[0065] Subsequently, the normalized original log data is further analyzed and identified, and abnormal log data is screened out by establishing a suitable model or algorithm. These abnormal log data may contain information related to system failures (especially effective and accurate identification of log data related to abnormal stack features and network failure features); finally, the abnormal log data is further analyzed, especially the abnormal stack information (stack trace) is identified and the source of the failure is determined. The abnormal stack information usually contains important details such as error information and call relationships during program execution, which can help locate the specific cause and location of the system failure; however, the cloud platform cluster network security log data failure prediction and processing method adopted in the embodiment of the present invention can efficiently predict and identify system failures by processing and analyzing the log data of the cloud data computing system, thereby improving the operational safety of the system. By real-time monitoring and analysis of log data, timely measures can be taken to repair or optimize before the failure occurs, reducing the impact of the failure on system services and users, and improving the availability and stability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0067] Figure 1 The present invention provides a cloud platform cluster network security log data fault prediction and processing method combined with the ELK platform to realize the cluster log collection and processing principle;

[0068] Figure 2 A schematic diagram of the operating steps of a cloud platform cluster network security log data fault prediction and processing method provided by an embodiment of the present invention;

[0069] Figure 3 A schematic diagram of the steps for filtering out abnormal log data in a cloud platform cluster network security log data fault prediction and processing method provided by an embodiment of the present invention;

[0070] Figure 4 A schematic diagram of the operational steps for obtaining a first classification result in a cloud platform cluster network security log data fault prediction and processing method provided by an embodiment of the present invention;

[0071] Figure 5 A schematic diagram of the conditional judgment steps for starting the third-level weight vector introduction operation in a cloud platform cluster network security log data fault prediction and processing method provided by an embodiment of the present invention;

[0072] Figure 6 A schematic diagram of the operation steps for obtaining a second classification result in a cloud platform cluster network security log data fault prediction and processing method provided by an embodiment of the present invention;

[0073] Figure 7 A schematic diagram of the operational steps for constructing a third-level fault weight vector in a cloud platform cluster network security log data fault prediction and processing method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0074] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0075] The present invention will be further described in detail below through specific embodiments in conjunction with the accompanying drawings.

[0076] like Figure 2 As shown, the present invention proposes a cloud platform cluster network security log data fault prediction and processing method, including the following steps:

[0077] Step S10: Obtaining original log data;

[0078] Step S20: normalizing the stack nodes executed by the application server cluster nodes corresponding to the original log data and the corresponding network fault log data to obtain normalized original log data;

[0079] Step S30: Analyze and identify the normalized original log data, and filter out abnormal log data;

[0080] Step S40: Identify the abnormal stack information of the abnormal log data to determine the source of the fault.

[0081] It should be noted that during the software operation, unexpected errors or abnormal situations may occur, causing the program to crash or malfunction. These abnormal situations will be recorded in the so-called exception log. In order to resolve these abnormal situations, it is necessary to analyze and process the exception log data, and one of the important steps is to identify the exception stack information.

[0082] Exception stack information usually includes a series of function call trace information, which is arranged in the order of calls and provides detailed information such as the specific location of the program exception and the number of code lines. This is very helpful for locating and resolving exceptions. Therefore, identifying and determining exception stack information in exception log data means extracting the exception stack information from the exception log and parsing and analyzing it to find the specific cause of the exception and then make repairs and improvements.

[0083] The above-mentioned embodiment of the present application first obtains the original log data of the cloud data computing system. These data contain various records and information of the system during operation. Then, for the original log data, normalization processing is performed, and log data of different formats and different types are uniformly converted into the same data format and unit for subsequent analysis and processing; then the normalized original log data is further analyzed and identified, and abnormal log data is screened out by establishing a suitable model or algorithm. These abnormal log data may contain information related to system failures (especially effective log data related to abnormal stack features and network failure features are accurately identified); finally, the abnormal log data is further analyzed, especially for identifying and determining the source of the failure for abnormal stack information (stack trace). The abnormal stack information usually contains important details such as error information and call relationships during program execution, which can help locate the specific cause and location of the system failure; however, the cloud platform cluster network security log data failure prediction processing method adopted in the embodiment of the present invention can efficiently predict and identify system failures by processing and analyzing the log data of the cloud data computing system, thereby improving the operational safety of the system. By real-time monitoring and analysis of log data, timely measures can be taken to repair or optimize before a failure occurs, reducing the impact of the failure on system services and users, and improving system availability and stability.

[0084] Specifically, in step S20, normalization processing is performed on the stack nodes executed by the application server cluster nodes corresponding to the original log data and the corresponding network fault log data to obtain normalized original log data, including the following steps:

[0085] Step S21: normalizing the stack nodes executed by the application server cluster nodes corresponding to the original log data to obtain normalized log data (i.e., the normalized original log data); normalizing the network fault log data corresponding to the original log data to obtain normalized log data; the normalized log data includes the normalized stack node original log data set and the normalized network fault log data set;

[0086] The final normalized stack node original log data set Wx*={w1*,w2*,…wi*,…wN*} is obtained, where N represents the number of total factors of the normalized stack node original log data set, and wi* is the normalized stack node original log data of the i-th stack node; the network fault log data set Wy*={w1*,w2*,…wi*,…wN*}, where N represents the number of total factors of the normalized network fault original log data set, and wi* is the normalized network fault original log data of the i-th network fault;

[0087] It should be noted that the technical solution of this application first calls multiple typical fault samples (i.e., samples of typical faults containing abnormal log data) from historical data; this application calls the above m samples (i.e., typical fault log samples), and the above m samples contain corresponding k features, and we hope to filter abnormal log data based on factors such as m samples, the jth network fault, and the nth abnormal stack fault. First, we can construct an m×k matrix A (i.e., the target matrix) to represent the log data after matrix operation processing, where each row represents a sample and each column represents a feature; the specific technical solution is as follows:

[0088] Specifically, if Figure 3 As shown, in step S30, the normalized original log data is analyzed and identified, and abnormal log data is obtained by screening, including the following steps:

[0089] Step S31: During the initialization process, the normalized original log data is analyzed and identified for the first time to obtain a first classification result; abnormal log data is screened from the first classification result;

[0090] Step S32: verifying the first classification result to obtain a verification result; determining whether the verification result meets the target expansion condition; if so, starting the third-level weight vector introduction operation;

[0091] Step S33: performing a second analysis and identification on the normalized original log data in combination with the third-level weight vector to obtain a second classification result; and filtering the second classification result to obtain abnormal log data.

[0092] It should be noted that the normalized log data is first classified to obtain a first classification result. By using an appropriate model or algorithm, the log data can be quickly and accurately classified and abnormal log data can be screened out. This abnormal log data may contain information about system failures. The first classification result is further verified to obtain a verification result, and then determine whether the target expansion conditions are met. If the verification result does not meet the expected conditions, the third-level weight vector introduction operation is not performed. Otherwise, the next operation is performed. Finally, the normalized original log data is combined with the third-level weight vector to perform a second classification to obtain the second classification result.

[0093] The third-level weight vector models and analyzes the correlations between different attributes or features in log data, enabling more comprehensive classification and identification of log data. The second-level classification results can more accurately screen out abnormal log data, improving the accuracy and efficiency of fault identification. By introducing the third-level weight vector, the correlations between attributes in log data can be more comprehensively considered, enhancing the accuracy of fault diagnosis. This method can help system administrators promptly identify and resolve potential faults, improving the reliability and stability of cloud data computing systems. This method can efficiently classify and identify cloud data computing system log data, thereby predicting and diagnosing system faults.

[0094] Specifically, if Figure 4 As shown, in step S31, during the initialization process, the normalized original log data is analyzed and identified for the first time to obtain a first classification result, including the following steps:

[0095] Step S311: Obtain historical data, select m typical fault log samples from the historical data to construct a typical fault log sample set M = {m1, m2, m3, ...mi}; wherein mi represents the i-th typical fault log sample; and obtain j feature vectors corresponding to each of the typical fault log samples to construct a feature vector set K = {k1, k2, k3, ...kj}; wherein kj is the j-th feature vector; these typical fault log samples should represent the situation where typical faults contain abnormal log data.

[0096] Calling servers of multiple cloud data computing systems that have failed, obtaining historical data (i.e., all historical log samples, including normal samples and original log samples of typical failures) from the servers of the multiple cloud data computing systems, and screening multiple typical failure log samples from the historical data;

[0097] Step S312: Construct a target matrix A based on all the typical fault log sample sets and all the feature vector sets K:

[0098] Among them, mikj is the jth eigenvector of the i-th typical fault sample log;

[0099] Step S313: Setting a network fault weight vector W = {w1, w2, wj . . . , wt} for each of the feature vectors; wherein wj is the weight of the j-th feature vector;

[0100] It should be noted that the above network fault weight vector is a first-level correlation feature: Assuming that the jth network fault is related to the first t features, a network fault weight vector W of size k is constructed. W = {w1, w2, wt..., wk}; where wt represents the weight of the tth feature. If the feature is not related to the network fault, the corresponding weight is 0.

[0101] Step S314: constructing a stack fault weight vector N={n1, n2, ..., nj, ng} for each of the feature vectors; wherein nj is the weight of the j-th feature in the exception stack;

[0102] It should be noted that the stack fault weight vectors described above are second-level correlation features: Assuming the nth abnormal stack is correlated with the next q features, a stack fault weight vector N of size k is constructed. N = {n1, n2, ...nq, nk}, where nq represents the weight of the qth feature in the abnormal stack. If the feature is not related to the abnormal stack, the corresponding weight is 0.

[0103] Step S315: Calculate and obtain a fault feature vector F based on the network fault weight vector W and the stack fault weight vector N;

[0104] The fault feature vector F is calculated as follows:

[0105] F=W+N;

[0106] Among them, W+N is the addition of corresponding elements;

[0107] It should be noted that, in the above embodiment of the present application, the fault feature vector is obtained by element-wise addition of the network fault weight vector w and the stack fault weight vector n.

[0108] Fault feature vector F = W + N;

[0109] Wherein, W+N represents the addition of corresponding elements to obtain the fault feature vector F.

[0110] Step S316: Calculate and obtain a comprehensive fault classification vector Y corresponding to each typical fault log sample based on the target matrix A and the fault feature vector F; input the comprehensive fault classification vector Y into a pre-trained classifier to output a first classification result;

[0111] The calculation method of the comprehensive fault classification vector Y is:

[0112] Y = F × A = {y1, y2, y3, ...yi, ...ym}; where yi is the comprehensive fault classification vector of the i-th typical fault log sample;

[0113] The first classification result is used to indicate whether the current i-th sample has a risk;

[0114] It should be noted that, finally, we can calculate the product of matrix A and the fault feature vector f to obtain a vector Y = {y1, y2, y3, ...yi, ...ym} of length m, where yi represents the degree of correlation between the i-th sample and the (comprehensive fault). The yi value is input into the classifier to obtain the first classification result. Based on this first classification result, we can filter the fault log data for the next step of fault classification and identification. The first classification result is used to indicate whether the current i-th sample has the risk of (comprehensive fault).

[0115] By analyzing historical data from multiple cloud data computing systems, we extract typical fault log samples and construct corresponding feature vectors and target matrices based on these samples. Furthermore, by setting network fault weight vectors and stack fault weight vectors, we can more accurately reflect the characteristics of the fault log samples related to network and stack faults. Finally, by multiplying the fault feature vector and the target matrix, we obtain a comprehensive fault classification vector, which can be used to determine whether the current sample has a fault risk.

[0116] This method can improve the accuracy and efficiency of fault diagnosis, helping to quickly discover and resolve system faults, and enhancing system reliability and stability. Furthermore, it can classify fault samples, helping users determine fault risks and take timely action to address them.

[0117] See also Figure 1The method adopted in the embodiment of the present application is combined with the ELK platform 100, which is a centralized log processing solution. It uses open source tools such as the ElasticSearch module 101, the Logstash module 102, the Elasticsearch cluster module 103 and the Kiabana module 104 to meet the query, sorting and statistical needs of large-scale cluster logs.

[0118] The ELK platform 100 (i.e., the main controller) uses the logs of the acquisition module (ElasticSearch module 101) for centralized management in the Logstash module 102. The logs are then formatted (Logstash module 102) and normalized and output to the Elasticsearch cluster module 103; the log data is then indexed by the storage path and the location is locked in the ElasticSearch module 101, and recorded in the cluster node log data chart, that is, the Logstash module collects the original log data generated by the acquisition module (ElasticSearch module 101), and then stores it in the cluster node managed by Elasticsearch, and also performs operations such as formatting (Logstash module 102) and normalization; after finally identifying and locking the target fault, the storage location of the fault log data can be locked by querying the cluster node log data chart, and the log file at the storage location can also be directly called for further analysis;

[0119] However, after determining the target fault, the Kiabana module 104 finally queries the data from the cluster nodes managed by Elasticsearch to generate a cluster node log data chart, and then returns it to the main controller (which then directly calls the log file in the storage location for further analysis).

[0120] Specifically, if Figure 5 As shown, in step S32, it is determined whether the verification result meets the target expansion condition. If so, the third-level weight vector introduction operation is started, including the following operation steps:

[0121] Step S321: pre-setting a secondary evaluation threshold; the secondary evaluation threshold includes an error rate standard threshold and a stack fault evaluation standard threshold;

[0122] Step S322: using the secondary evaluation threshold to verify the first classification result of the cloud data computing system to obtain a verification result;

[0123] It should be noted that the above-mentioned verification method of verifying the classification result of the cloud data computing system using the secondary evaluation threshold includes: calculating and obtaining an overall classification error rate based on the first classification result; determining whether the overall classification error rate in the current first classification result of a preset number of original log data is higher than the error rate standard threshold; if it is higher than the error rate standard threshold, further screening multiple abnormal log data that actually have stack faults;

[0124] Count the number of first classification results of abnormal log data that incorrectly predict the presence of a stack abnormality, and divide the number of first classification results of abnormal log data that incorrectly predict the presence of a stack abnormality by the ratio of the total number of samples of abnormal log data that actually have a stack fault, thereby obtaining a stack fault judgment error rate (i.e., the ratio of the number of samples of the first classification results of abnormal log data that are incorrectly predicted to the number of samples of abnormal log data that actually have a stack fault); continue to determine whether the stack fault judgment error rate is greater than the stack fault evaluation standard threshold;

[0125] Step S323: determining whether the verification result meets a preset verification error rate condition; if so, starting the third-level weight vector introduction operation;

[0126] The verification error rate condition refers to: the overall classification error rate is higher than the error rate standard threshold and the stack fault judgment error rate is higher than the stack fault evaluation standard threshold;

[0127] The above-mentioned target expansion conditions refer to the secondary judgment thresholds, namely the error rate standard threshold and the stack fault evaluation standard threshold; a comprehensive judgment is made through the overall classification error rate and the stack fault judgment error rate. When the overall classification error rate in the current first classification result of a preset number of original log data is judged to be higher than the error rate standard threshold, and when it is judged whether the stack fault judgment error rate is greater than the stack fault evaluation standard threshold, a third-level weight vector operation is performed to improve the accuracy of fault prediction.

[0128] It should be noted that, in a further technical solution of the above-mentioned embodiment of the present application, the first classification result is verified to obtain a verification result; it is determined whether the verification result meets the preset target expansion conditions, and if so, the third-level weight vector introduction operation is started to control the third-level weight vector introduction operation to classify the sample; if not, the above-mentioned step S40 is directly performed to identify the abnormal stack information of the abnormal log data to determine the source of the fault.

[0129] In the above-mentioned embodiment of the present application, since the fault prediction capability is seriously inconsistent with the actual situation, that is, the first classification result obtained may cause load abnormality due to reasons such as access volume, which may further lead to misjudgment and identification of stack failure or network failure, resulting in a huge error in the classification result. Therefore, it is necessary to introduce a third-level weight vector, that is, to introduce a third-level weight factor, so as to obtain a more accurate second classification result, thereby ultimately improving the accuracy of fault prediction.

[0130] Preferably, as an implementable method, after the third-level weight vector operation is introduced, the correction and identification processing operation of the abnormal log data still needs to be repeatedly performed, as detailed in the following processing process.

[0131] It should be noted that; the study found that in detecting special fault scenarios, the researchers found that: if there are certain special fault scenarios, and the model performs poorly in these scenarios, it may be because there is a third-pole important influencing factor that leads to the frequent occurrence of faults, but this third-pole factor has not been taken into consideration. For example: under high access volume, the failure rate is high, but there is a misjudgment problem, and the load failure is misjudged as a stack failure. Therefore, high access volume at this time may be the third-pole factor that causes load failures, and is an important influencing factor that leads to the misjudgment of stack failures; therefore, the embodiment of the present invention has an in-depth understanding and analysis of special fault scenarios, which can not only identify shallow fault scenarios, but also further discover other fault scenarios that may be misjudged as stack failures, and classify and analyze them. Therefore, judging whether to introduce a third-pole weight vector is an important deep-level discovery and processing operation.

[0132] Specifically, if Figure 6 As shown, in S33, the normalized original log data is analyzed and identified for the second time in combination with the third-level weight vector to obtain a second classification result, which specifically includes:

[0133] Step S331: Obtain e extended feature vectors K'={k'1, k'2, k'3, ..., k'e} corresponding to the typical fault log sample; wherein k'e is the e-th extended feature vector; construct an extended target matrix A' based on the extended feature K' and the typical fault log sample set M;

[0134] The expanded target matrix A' is expressed as:

[0135] Among them, mik'e is the e-th extended feature vector of the i-th typical fault sample log;

[0136] It should be noted that the above embodiment of the present application first constructs a target matrix: the e extended features contained in the i samples are combined into an m×k' matrix A' (i.e., the expanded target matrix), where each row represents a sample and each column represents a feature; that is, the extended feature parameter of the visit volume is reintroduced into it;

[0137] Step S332: presetting an extended network fault weight vector W'={w'1, w'2, w'j..., w't} for each extended feature vector; wherein w'j is the weight of the j-th extended feature vector in the network fault;

[0138] It should be noted that the above-mentioned extended network fault weight vector is the first-level correlation feature. Assuming that the j-th network fault is related to the first t features, a network fault weight vector W' of size t is constructed. If the extended feature is not related to the network fault, the corresponding weight is 0.

[0139] Step S333: constructing an extended stack fault weight vector N'={n'1, n'2, ... n'j, n'g} for each extended feature vector; wherein n'j is the weight of the j-th extended feature vector in the exception stack;

[0140] It should be noted that the above-mentioned extended stack fault weight vector is a second-level correlation feature. Assuming that the j-th abnormal stack is related to the next g features, a stack fault weight vector N' of size g is constructed. If the feature is not related to the abnormal stack, the corresponding weight is 0.

[0141] Step S334: constructing a third-level fault weight vector P = {p1, p2, .. pj, pf} for each of the extended feature vectors; wherein pf is the weight of the f-th extended feature vector in the third-level fault;

[0142] It should be noted that the third-level fault weight vector is a visit weight vector. Assuming that the jth visit is related to the next f features, a visit weight vector P of size f is constructed. If the extended feature is not related to the visit, the corresponding weight is 0.

[0143] Step S335: Calculate and obtain an extended fault feature vector F' based on the extended network fault weight vector W', the extended stack fault weight vector N', and the third-level fault weight vector P (or the weight of the access weight vector mentioned above);

[0144] The calculation method of the extended fault feature vector F' is:

[0145] F'=W'+N'+P;

[0146] Among them, W'+N'+P is the addition of corresponding elements;

[0147] It should be noted that, in the above embodiment of the present application, the extended fault feature vector is obtained by element-wise addition of the extended network fault weight vector W', the extended stack fault weight vector N', and the third-level fault weight vector P.

[0148] Step S336: Calculate and obtain the expanded comprehensive fault classification vector Y' corresponding to each typical fault log sample based on the expanded target matrix A' and the expanded fault vector F'; input the expanded comprehensive fault classification vector Y' into the pre-trained classifier to output a second classification result;

[0149] The calculation method of the extended comprehensive fault classification vector Y' is:

[0150] Y'=F'×A'={y'1, y'2, y'3, ...y'i, ...y'm}; where y'i is the comprehensive fault classification vector of the i-th typical fault log sample;

[0151] The second classification result is used to indicate whether the current i-th sample has a risk;

[0152] It should be noted that in the above technical solution, when considering the third-level fault factors, the introduction of the third-level weight vector P can further improve the performance of the model; the introduction of the third-level weight vector P leads to the expansion of the weight parameters in three dimensions; the role of the third-level weight vector P is: the third-level weight vector P is a vector of size k, representing the third-level weight; the introduction of P is to consider new fault factors that may emerge in the extensive development of sample data, making the model more flexible;

[0153] The third-level weight vector P in the above-mentioned embodiment of the present application can determine its corresponding weight vector P through expert knowledge, feature importance analysis, etc.; if there is new abnormal log data, P can be updated through supervised learning or semi-supervised learning; the above-mentioned third-level weight vector is P = {p1, p2, ..pj, pf}; after the third-level weight vector is introduced: it is necessary to perform an expansion processing operation of the target matrix: construct an expanded target matrix A', a fault feature vector F' which includes network fault weights, stack fault weights and the introduced third-level weights (for example, the weight of the impact of access volume on the occurrence of fault logs), and then perform a product calculation of the feature vector and the expanded target matrix to obtain an expanded comprehensive fault classification vector Y', so as to ensure that in subsequent operations, multi-dimensional factors leading to the generation of fault logs can be considered, so as to more accurately identify and analyze the fault logs, and then more quickly and accurately identify the fault location and perform maintenance processing.

[0154] Regarding the issues related to log data processing, researchers found that the main problems with log data are reflected in the following three aspects: first, the information in the log data is unclear, resulting in: reduced execution efficiency (the corresponding positions of the application server cluster nodes in the embodiment of the present application are already clear); second, the format of the log data is not standardized. This results in: it is not convenient to read key information in a targeted manner, and it is not possible to collect it conveniently (the method adopted in this embodiment is to normalize the stack nodes executed by the corresponding application server cluster nodes and the corresponding network fault log data); third, there is little log data and a lack of key information (i.e., the lack of key features that affect the occurrence of faults). This results in: reduced efficiency in locating problems (the method adopted in this embodiment of the present application is to continuously update, iterate, and dynamically adjust and promptly discover new key features that affect the occurrence of faults, thereby smoothly introducing third-party weighting factors to solve this problem);

[0155] Specifically, if Figure 7 As shown, in step S334, a third-level fault weight vector P={p1, p2, .. pj, pf} is constructed for each of the extended feature vectors, including the following steps:

[0156] Step S3341: Obtain the third-level correlation feature corresponding to each of the typical fault log samples (the third-level correlation feature is generally pageview data);

[0157] Step S3342: normalizing the third-level correlation features to obtain normalized correlation feature data;

[0158] Step S3343: performing statistics on the normalized correlation feature data to obtain distribution information corresponding to the third-level correlation feature;

[0159] Step S3344: assigning weights to each of the typical fault log samples based on the distribution information to obtain a third-level fault weight vector corresponding to each of the extended feature vectors;

[0160] It should be noted that the above-mentioned embodiment of the present application first extracts third-level correlation features, such as access volume data, from typical fault log samples. Third-level correlation features are usually related to the occurrence and impact of faults. By collecting these features, the context information of the fault event can be more comprehensively described; further normalization processing helps to convert third-level correlation features of different ranges and units into the same scale, so that different features can be better compared and processed in the subsequent statistics and weight calculation process; by statistically analyzing the distribution information of the third-level correlation features, it helps to understand the overall distribution of these features, such as statistics such as mean and variance; this information can provide a basis for subsequent weight calculations, making the weights more in line with actual conditions; by assigning weights to each typical fault log sample based on distribution information, the importance of each sample in constructing the fault weight vector can be more reasonably reflected; for constructing extended feature vectors, these weights can be used for weighted averaging to emphasize the influence of certain key features.

[0161] Normalization and statistical distribution information provide more comprehensive and comparable feature information for subsequent weight calculations. Weights are assigned based on the relative position of samples in the overall distribution, allowing more representative or influential samples to have a greater impact on the final weight when constructing the fault weight vector. This approach makes the fault weight vector more adaptable to actual conditions, better reflecting the influence of each sample in the entire system, and improving the model's ability to predict and diagnose actual faults.

[0162] The embodiments of the present application adopt a technical solution that, by viewing log files and analyzing text features, it is noted that this analysis process has a feedback and iteration function, and the log data recording the exception stack can be found and the exception information can be obtained from it. The above-mentioned exception stack typically contains the type of exception, the cause of the error, the location on the call stack, and related information. Developers can track the source of the problem based on the exception stack information and perform appropriate debugging and repair.

[0163] In summary, the present invention proposes a method for predicting and processing network security log data faults in a cloud platform cluster. The method first obtains raw log data containing various records and information during system operation, then normalizes the raw log data; further analyzes and identifies rows in the normalized raw log data to filter out abnormal log data; and finally further analyzes the abnormal log data, particularly identifying and determining the source of the fault based on the abnormal stack information.

[0164] Specifically, in the process of filtering out abnormal log data, the normalized log data is first classified to obtain the first classification result. The first classification result is further verified to obtain the verification result, and it is determined whether the target expansion condition is met. If the verification result does not meet the expected condition, the third-level weight vector is not introduced, and the abnormal log data is directly filtered out from the first classification result. If the verification result meets the expected condition, the third-level weight vector is introduced, and finally, the normalized original log data is classified for the second time in combination with the third-level weight vector to obtain the second classification result. The third-level weight vector can more comprehensively classify and identify log data, so the second classification result can more accurately filter out abnormal log data, improving the accuracy and efficiency of fault identification. It can efficiently classify and identify cloud data computing system log data, thereby predicting and diagnosing system faults.

[0165] During the first classification process, historical data from multiple cloud data computing systems is analyzed to extract typical fault log samples. Based on these samples, corresponding feature vectors and target matrices are constructed. Furthermore, network and stack fault weight vectors are set to more accurately reflect the characteristics of the fault log samples related to network and stack faults. Finally, by multiplying the fault feature vector and the target matrix, a comprehensive fault classification vector is obtained, which can be used to determine whether the current sample has a fault risk.

[0166] This method can improve the accuracy and efficiency of fault diagnosis, helping to quickly discover and resolve system faults, and enhancing system reliability and stability. Furthermore, it can classify fault samples, helping users determine fault risks and take timely action to address them.

[0167] At the same time, when determining whether to introduce a third-level weight vector, the fault prediction capability is seriously inconsistent with the actual situation. That is, the first classification result may cause load abnormalities due to factors such as access volume, which may further lead to the identification of stack faults or network faults, resulting in a large error in the classification result. Therefore, it is necessary to introduce a third-level weight vector to obtain a more accurate second classification result, thereby ultimately improving the accuracy of fault prediction.

[0168] Specifically, in the process of analyzing and obtaining the second classification results, the performance of the model can be further improved by introducing the third-level weight vector P while considering the third-level fault factors.

[0169] The introduction of the third-level weight vector P leads to a three-dimensional expansion of the weight parameters. The third-level weight vector P is a vector of size k, representing the third-level weight. The introduction of P is to consider new failure factors that may emerge in the extensive development of sample data, making the model more flexible.

[0170] Furthermore, in the process of constructing the third-level weight vector P, third-level correlation features, such as access data, are first extracted from typical fault log samples. Third-level correlation features are usually related to the occurrence and impact of faults. By collecting these features, the context information of the fault event can be more comprehensively described. Further normalization processing helps to convert third-level correlation features of different ranges and units into the same scale, so that different features can be better compared and processed in the subsequent statistics and weight calculation process. By statistically analyzing the distribution information of third-level correlation features, it is helpful to understand the overall distribution of these features, such as statistics such as mean and variance. This information can provide a basis for subsequent weight calculations, making the weights more consistent with actual conditions. By assigning weights to each typical fault log sample based on the distribution information, the importance of each sample in constructing the fault weight vector can be more reasonably reflected. For the construction of the extended feature vector, these weights can be used for weighted averaging to emphasize the influence of certain key features.

[0171] Normalization and statistical distribution information provide more comprehensive and comparable feature information for subsequent weight calculations. Weights are assigned based on the relative position of samples in the overall distribution, allowing more representative or influential samples to have a greater impact on the final weight when constructing the fault weight vector. This approach makes the fault weight vector more adaptable to actual conditions, better reflecting the influence of each sample in the entire system, and improving the model's ability to predict and diagnose actual faults.

[0172] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. A person skilled in the art may modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cloud platform cluster network security log data fault prediction and processing method, characterized in that: The steps are as follows: Get raw log data; Normalizing the stack nodes executed by the application server cluster nodes and the corresponding network fault log data corresponding to the original log data to obtain normalized original log data; Analyze and identify the normalized original log data and filter out abnormal log data; Identify the exception stack information in the exception log data to determine the source of the fault.

2. A cloud platform cluster network security log data fault prediction and processing method according to claim 1, characterized in that: The normalized original log data includes a normalized stack node original log data set and a normalized network fault log data set.

3. A cloud platform cluster network security log data fault prediction and processing method according to claim 2, characterized in that: Normalizing the stack nodes executed by the application server cluster nodes corresponding to the original log data and the corresponding network fault log data to obtain the normalized original log data includes the following steps: Normalization processing is performed on the stack nodes executed by the application server cluster nodes corresponding to the original log data to obtain normalized log data; and normalization processing is performed on the network fault log data corresponding to the original log data to obtain normalized log data.

4. A cloud platform cluster network security log data fault prediction and processing method according to claim 3, characterized in that: The analysis and identification of the normalized original log data to obtain abnormal log data includes the following steps: During the initialization process, the normalized original log data is first analyzed and identified to obtain a first classification result; abnormal log data is screened from the first classification result; Verifying the first classification result to obtain a verification result; determining whether the verification result meets the target expansion condition, and if so, initiating a third-level weight vector introduction operation; Performing a second analysis and identification on the normalized original log data in combination with the third-level weight vector to obtain a second classification result; Abnormal log data is obtained by screening from the second classification results.

5. A cloud platform cluster network security log data fault prediction and processing method according to claim 4, characterized in that: During the initialization process, the normalized original log data is analyzed and identified for the first time to obtain a first classification result, including the following steps: Obtain historical data, select m typical fault log samples from the historical data to construct a typical fault log sample set M = {m1, m2, m3, ...mi}; where mi represents the i-th typical fault log sample; and obtain j feature vectors corresponding to each of the typical fault log samples to construct a feature vector set K = {k1, k2, k3, ...kj}; where kj is the j-th feature vector; Construct a target matrix A based on all the typical fault log sample sets and all the feature vector sets K: Among them, mikj is the jth eigenvector of the i-th typical fault sample log; Setting a network fault weight vector W = {w1, w2, wj..., wt} for each of the feature vectors; wherein wj is the weight of the j-th feature vector; For each of the feature vectors, a stack fault weight vector N = {n1, n2, ..., nj, ng} is constructed; wherein nj is the weight of the j-th feature in the exception stack; Obtaining a fault feature vector F by calculation based on the network fault weight vector W and the stack fault weight vector N; The fault feature vector F is calculated as: F = W + N; where W + N is the sum of corresponding elements; Calculate and obtain a comprehensive fault classification vector Y corresponding to each typical fault log sample based on the target matrix A and the fault feature vector F; input the comprehensive fault classification vector Y into a pre-trained classifier to output a first classification result; The calculation method of the comprehensive fault classification vector Y is: Y = F × A = {y1, y2, y3, ...yi, ...ym}; where yi is the comprehensive fault classification vector of the i-th typical fault log sample; The first classification result is used to indicate whether the current i-th sample has a risk.

6. A cloud platform cluster network security log data fault prediction and processing method according to claim 5, characterized in that: The step of determining whether the verification result meets the target expansion condition and, if so, starting the third-level weight vector introduction operation includes the following steps: Presetting a secondary evaluation threshold; using the secondary evaluation threshold to verify the first classification result of the cloud data computing system to obtain a verification result; Determine whether the verification result meets the preset verification error rate condition, and if so, start the third-level weight vector introduction operation; The verification error rate condition refers to: the overall classification error rate is higher than the error rate standard threshold and the stack fault judgment error rate is greater than the stack fault evaluation standard threshold.

7. A cloud platform cluster network security log data fault prediction and processing method according to claim 6, characterized in that: The secondary evaluation threshold includes an error rate standard threshold and a stack fault evaluation standard threshold.

8. A cloud platform cluster network security log data fault prediction and processing method according to claim 7, characterized in that: The step of verifying the first classification result of the cloud data computing system by using the secondary evaluation threshold to obtain a verification result includes the following steps: Calculating an overall classification error rate based on the first classification result; determining whether the overall classification error rate is higher than an error rate standard threshold; and if so, further screening multiple abnormal log data indicating actual stack failures; Counting the number of abnormal log data with actual stack failures, and calculating the ratio of the number of samples of the first classification results of the abnormal log data with predicted errors to the abnormal log data with actual stack failures; Determine whether the stack fault judgment error rate is greater than the stack fault evaluation standard threshold.

9. A cloud platform cluster network security log data fault prediction and processing method according to claim 8, characterized in that: The second analysis and identification of the normalized original log data is performed in combination with the third-level weight vector to obtain a second classification result, which specifically includes: Obtain e extended feature vectors K'={k'1, k'2, k'3, ..., k'e} corresponding to the typical fault log sample; wherein k'e is the e-th extended feature vector; construct an extended target matrix A' based on the extended feature K' and the typical fault log sample set M; The expanded target matrix A' is expressed as: Among them, mik'e is the e-th extended feature vector of the i-th typical fault sample log; Presetting an extended network fault weight vector W'={w'1, w'2, w'j..., w't} for each extended feature vector; wherein w'j is the weight of the j-th extended feature vector in the network fault; Constructing an extended stack fault weight vector N'={n'1, n'2, ... n'j, n'g} for each extended feature vector; wherein n'j is the weight of the j-th extended feature vector in the exception stack; Constructing a third-level fault weight vector P = {p1, p2, .. pj, pf} for each of the extended feature vectors; wherein pf is the weight of the f-th extended feature vector in the third-level fault; Obtain an extended fault feature vector F' based on the extended network fault weight vector W', the extended stack fault weight vector N', and the third-level fault weight vector P; The calculation method of the extended fault feature vector F' is: F'=W'+N'+P; Among them, W'+N'+P is the addition of corresponding elements; Calculating and obtaining an expanded comprehensive fault classification vector Y' corresponding to each of the typical fault log samples based on the expanded target matrix A' and the expanded fault vector F'; inputting the expanded comprehensive fault classification vector Y' into a pre-trained classifier to output a second classification result; The calculation method of the extended comprehensive fault classification vector Y' is: Y'=F'×A'={y'1, y'2, y'3, ...y'i, ...y'm}; where y'i is the comprehensive fault classification vector of the i-th typical fault log sample; The second classification result is used to indicate whether the current i-th sample has a risk.

10. A cloud platform cluster network security log data fault prediction and processing method according to claim 9, characterized in that: The step of constructing a third-level fault weight vector P={p1, p2, .. pj, pf} for each of the extended feature vectors includes the following steps: Obtaining the third-level correlation features corresponding to each of the typical fault log samples; Normalizing the third-level correlation features to obtain normalized correlation feature data; Performing statistics on the normalized correlation feature data to obtain distribution information corresponding to the third-level correlation feature; A weight is assigned to each of the typical fault log samples based on the distribution information to obtain a third-level fault weight vector corresponding to each of the extended feature vectors.

Images