Cloud platform cluster network security log data fault prediction processing method

Abstract

The application discloses a kind of cloud platform cluster network security log data fault prediction processing methods, first, the original log data of cloud data computing system is acquired, then for original log data, normalization processing is carried out;Subsequently, further analysis and identification are carried out to the original log data after normalization, and abnormal log data is screened out;Finally, further analysis is carried out to abnormal log data, especially for abnormal stack information, identify and determine fault source;By processing and analyzing the log data of cloud data computing system, system failure is efficiently predicted and identified, so as to improve the operation safety of system;By real-time monitoring and analyzing log data, measures can be taken in time to repair or optimize before failure occurs, reduce the influence of failure on system business and user, improve the availability and stability of system.

Description

Technical Field

[0001] This invention relates to the field of cloud computing, and in particular to a method for predicting and processing network security log data faults in cloud platform clusters. Background Technology

[0002] Cloud data computing systems have a wide range of applications. For example, they can be used in e-commerce back-end order matching systems or e-commerce platform statistical systems (such as distributed cluster systems).

[0003] Generally, server systems (such as cloud computing systems) are prone to system catastrophic failures during large-scale operational computing. These failures can be caused by a variety of factors, including network system crashes (network failures), system errors, application server cluster node crashes, and configuration errors. They can also be caused by system security issues such as intrusions, hacker attacks, and virus infections.

[0004] Therefore, monitoring text logs is crucial; a common approach is to monitor log files: set up monitoring programs to periodically check the status and content of log files to ensure they are recording correctly. Thus, a comprehensive inspection and monitoring of the logging system is necessary to promptly identify and resolve problems, ensuring the system's normal operation.

[0005] Monitoring log files currently exist in the following ways: Log data reflects the operating status of the system (server system or software application system). Analyzing the system's operating log data, extracting fault characteristics, and performing fault identification and processing are common methods for system fault diagnosis.

[0006] In particular, system disasters caused by network system crashes (network failures), system errors, application server cluster node crashes, and configuration errors are the most frequent. During certain periods of time (peak computing periods), servers may be prone to log failures caused by system errors, application server cluster node crashes, and configuration errors.

[0007] Further research revealed that various operational conditions can occur during system operation (e.g., application server cluster node crashes, configuration errors), resulting in different logs. In this case, extracting fault features from complex and variable log data and identifying fault logs is crucial. However, research found that most existing processing methods focus on log analysis for routine network faults. Conventional network fault identification constraint identification methods cannot fully identify whether log data is abnormal, which puts great pressure on log fault identification. Summary of the Invention

[0008] This invention proposes a method for predicting and processing network security log data faults in cloud platform clusters, comprising the following steps:

[0009] Obtain raw log data;

[0010] The stack traces executed by the application server cluster nodes corresponding to the original log data and the corresponding network fault log data are normalized to obtain the normalized original log data.

[0011] The normalized raw log data is analyzed and identified, and abnormal log data is filtered out.

[0012] Identify the source of the fault by analyzing the exception stack trace in the exception log data.

[0013] Preferably, the normalized raw log data includes a set of normalized stack node raw log data and a set of normalized network fault log data.

[0014] Preferably, the normalization process performed on the stack traces executed by the application server cluster nodes corresponding to the original log data and the corresponding network fault log data to obtain the normalized original log data includes the following steps:

[0015] The stack traces executed by the application server cluster nodes corresponding to the original log data are normalized to obtain normalized log data; the network fault log data corresponding to the original log data are also normalized to obtain normalized log data.

[0016] Preferably, the step of analyzing and identifying the normalized raw log data and filtering out abnormal log data includes the following steps:

[0017] During initialization, the normalized raw log data is analyzed and identified for the first time to obtain the first classification result; abnormal log data is then selected from the first classification result.

[0018] The first classification result is verified to obtain the verification result; it is determined whether the verification result meets the target expansion condition. If so, the weight factor introduction operation is initiated.

[0019] The normalized raw log data is analyzed and identified a second time by combining the weighting factors to obtain a second classification result; abnormal log data is then selected from the second classification result.

[0020] Preferably, during the initialization process, the normalized raw log data is analyzed and identified for the first time to obtain a first classification result, including the following steps:

[0021] Obtain historical data, select m typical fault log samples from the historical data to construct a typical fault log sample set M={m1, m2, m3, ... mi}; where mi represents the i-th typical fault log sample; and obtain j feature vectors corresponding to each typical fault log sample to construct a feature vector set K={k1, k2, k3, ... kj}; where kj is the j-th feature vector.

[0022] Construct a target matrix A based on all the typical fault log sample sets and all the feature vector sets K:

[0023] Where, mikj is the j-th feature vector of the i-th typical fault sample log;

[0024] A network fault weight vector W = {w1, w2, wj, ..., wt} is set for each of the aforementioned feature vectors; where wj is the weight of the j-th feature vector.

[0025] For each of the aforementioned feature vectors, a stack fault weight vector N = {n1, n2, ..., nj, ng} is constructed; where nj is the weight of the j-th feature in the fault stack.

[0026] The fault feature vector F is calculated and obtained based on the network fault weight vector W and the stack fault weight vector N;

[0027] The fault feature vector F is calculated as follows:

[0028] F = W + N;

[0029] Where W+N represents the sum of corresponding elements;

[0030] The comprehensive fault classification vector Y corresponding to each typical fault log sample is calculated based on the target matrix A and the fault feature vector F; the comprehensive fault classification vector Y is input into a pre-trained classifier to output the first classification result;

[0031] The comprehensive fault classification vector Y is calculated as follows:

[0032] Y = F × A = {y1, y2, y3, ... yi, ... ym}; where yi is the comprehensive fault classification vector of the i-th typical fault log sample;

[0033] The first classification result is used to indicate whether the current i-th sample is at risk;

[0034] Preferably, determining whether the verification result meets the target expansion condition, and if so, initiating the weight factor introduction operation, includes the following steps:

[0035] A secondary evaluation threshold is preset; the secondary evaluation threshold includes an error rate standard threshold and a stack fault evaluation standard threshold;

[0036] The first classification result of the cloud data computing system is verified using the secondary evaluation threshold to obtain the verification result;

[0037] Determine whether the verification result meets the preset verification error rate condition; if so, initiate the weight factor introduction operation.

[0038] The verification error rate condition refers to: the overall classification error rate is higher than the error rate standard threshold and the stack fault judgment error rate is greater than the stack fault evaluation standard threshold.

[0039] Preferably, the step of verifying the first classification result of the cloud data computing system using the secondary evaluation threshold to obtain the verification result includes the following steps:

[0040] The overall classification error rate is calculated based on the first classification result; it is then determined whether the overall classification error rate is higher than the error rate standard threshold. If so, multiple abnormal log data that actually have stack faults are further filtered.

[0041] The number of abnormal log data that actually have stack faults is counted, and the proportion of the number of samples of the first classification result of the abnormal log data that obtained the prediction error is relative to the number of abnormal log data that actually have stack faults is calculated; then it is determined whether the stack fault judgment error rate is greater than the stack fault evaluation standard threshold.

[0042] Preferably, the second analysis and identification of the normalized original log data, combined with the weighting factors, to obtain the second classification result specifically includes:

[0043] Obtain the e extended feature vectors K'={k'1, k'2, k'3, ..., k'e} corresponding to the typical fault log sample; where k'e is the e-th extended feature vector; construct the extended target matrix A' based on the extended feature K' and the typical fault log sample set M;

[0044] The extended target matrix A' is represented as follows:

[0045] Where, mik'e is the e-th extended feature vector of the log of the i-th typical fault sample;

[0046] For each of the extended feature vectors, a pre-defined extended network fault weight vector W'={w'1,w'2,w'j...,w't} is set; where w'j is the weight of the j-th extended feature vector in the network fault.

[0047] For each of the extended feature vectors, construct an extended stack fault weight vector N'={n'1,n'2,...n'j,n'g}; where n'j is the weight of the j-th extended feature vector in the exception stack;

[0048] Construct a third-level fault weight vector P={p1,p2,..pj,pf} for each of the extended feature vectors; where pf is the weight of the f-th extended feature vector in the third-level fault.

[0049] The extended network fault weight vector W', the extended stack fault weight vector N', and the third-level fault weight vector P (or the weight of the access volume weight vector mentioned above) are used to calculate and obtain the extended fault feature vector F'.

[0050] The extended fault feature vector F' is calculated as follows:

[0051] F' = W' + N' + P;

[0052] Where W'+N'+P is the sum of corresponding elements;

[0053] The extended comprehensive fault classification vector Y' corresponding to each typical fault log sample is calculated based on the extended target matrix A' and the extended fault vector F'; the extended comprehensive fault classification vector Y' is input into a pre-trained classifier, and the second classification result is output.

[0054] The extended comprehensive fault classification vector Y' is calculated as follows:

[0055] Y'=F'×A'={y'1,y'2,y'3,...y'i,...y'm};where y'i is the comprehensive fault classification vector of the i-th typical fault log sample;

[0056] The second classification result is used to indicate whether the current i-th sample is at risk.

[0057] Preferably, the step of constructing a third-level fault weight vector P={p1,p2,..pj,pf} for each of the extended feature vectors includes the following steps:

[0058] Obtain the third-level correlation features corresponding to each of the typical fault log samples;

[0059] The third-level association features are normalized to obtain normalized association feature data;

[0060] Statistical analysis is performed on the normalized correlation feature data to obtain the distribution information corresponding to the third-level correlation feature;

[0061] Based on the distribution information, weights are assigned to each of the typical fault log samples to obtain the third-level fault weight vector corresponding to each of the extended feature vectors.

[0062] Compared with the prior art, the embodiments of the present invention have at least the following technical advantages:

[0063] Analysis of the cloud platform cluster network security log data fault prediction and processing method provided by the present invention shows that, in specific applications:

[0064] First, the raw log data of the cloud data computing system is obtained. This data contains various records and information during the system's operation. Then, the raw log data is normalized to convert log data of different formats and types into the same data format and unit for subsequent analysis and processing.

[0065] Subsequently, the normalized raw log data undergoes further line analysis and identification. By establishing appropriate models or algorithms, abnormal log data is filtered out. This abnormal log data may contain information related to system failures (especially log data with accurate identification of abnormal stack trace characteristics and network failure characteristics). Finally, the abnormal log data is further analyzed, particularly identifying and determining the source of the failure based on the abnormal stack trace. Abnormal stack trace information typically contains important details such as error messages and call relationships during program execution, which can help pinpoint the specific cause and location of system failures. Furthermore, the cloud platform cluster network security log data failure prediction and processing method adopted in this embodiment of the invention can efficiently predict and identify system failures by processing and analyzing log data from cloud data computing systems, thereby improving the operational security of the system. By monitoring and analyzing log data in real time, timely measures can be taken to repair or optimize before failures occur, reducing the impact of failures on system services and users, and improving system availability and stability. Attached Figure Description

[0066] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0067] Figure 1 This invention provides a cloud platform cluster network security log data fault prediction and processing method that combines the ELK platform to achieve cluster log collection and processing.

[0068] Figure 2 This is a schematic diagram illustrating the operation steps of a cloud platform cluster network security log data fault prediction and processing method provided in an embodiment of the present invention;

[0069] Figure 3 This is a schematic diagram illustrating the steps for filtering abnormal log data in a cloud platform cluster network security log data fault prediction and processing method provided in an embodiment of the present invention.

[0070] Figure 4 This is a schematic diagram illustrating the steps for obtaining the first classification result in a cloud platform cluster network security log data fault prediction and processing method provided in an embodiment of the present invention.

[0071] Figure 5 This is a schematic diagram of the condition judgment step for initiating the weight factor introduction operation in a cloud platform cluster network security log data fault prediction and processing method provided in an embodiment of the present invention.

[0072] Figure 6 This is a schematic diagram illustrating the steps for obtaining the second classification result in a cloud platform cluster network security log data fault prediction and processing method provided in an embodiment of the present invention.

[0073] Figure 7 This is a schematic diagram illustrating the steps involved in constructing a third-level fault weight vector in a cloud platform cluster network security log data fault prediction and processing method provided in an embodiment of the present invention. Detailed Implementation

[0074] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0075] The present invention will now be described in further detail with reference to specific embodiments and accompanying drawings.

[0076] like Figure 2 As shown, this invention proposes a method for predicting and processing network security log data faults in a cloud platform cluster, comprising the following steps:

[0077] Step S10: Obtain raw log data;

[0078] Step S20: Normalize the stack nodes executed by the application server cluster nodes corresponding to the original log data and the corresponding network fault log data respectively to obtain the normalized original log data.

[0079] Step S30: Analyze and identify the normalized raw log data, and filter out abnormal log data;

[0080] Step S40: Identify the source of the fault by analyzing the exception stack information in the exception log data.

[0081] It should be noted that during the operation of the software, unexpected errors or abnormal situations may occur, causing the program to crash or fail to work properly. These abnormal situations will be recorded in the so-called exception log. In order to resolve these abnormal situations, the exception log data needs to be analyzed and processed, and one important step is to identify the exception stack information.

[0082] Exception stack traces typically include a series of function call traces arranged in the order of the calls, providing detailed information such as the specific location of the program exception and the line number of the code. This is very helpful in locating and resolving exceptions. Therefore, identifying and determining exception stack traces in exception log data involves extracting exception stack traces from exception logs, parsing and analyzing them to find the specific cause of the exception and then repairing and improving it.

[0083] The embodiments described above first acquire the raw log data of the cloud data computing system. This data contains various records and information during system operation. Then, the raw log data is normalized to convert log data of different formats and types into the same data format and unit for subsequent analysis and processing. Subsequently, the normalized raw log data is further analyzed and identified. By establishing appropriate models or algorithms, abnormal log data is filtered out. This abnormal log data may contain information related to system failures (especially log data with accurate identification of abnormal stack trace characteristics and network failure characteristics). Finally, the abnormal log data is further analyzed, particularly identifying and determining the source of the failure based on the abnormal stack trace information. Abnormal stack trace information typically contains important details such as error information and call relationships during program execution, which can help locate the specific cause and location of system failures. Furthermore, the cloud platform cluster network security log data failure prediction and processing method adopted in this embodiment can efficiently predict and identify system failures by processing and analyzing the log data of the cloud data computing system, thereby improving the operational security of the system. By monitoring and analyzing log data in real time, timely measures can be taken to repair or optimize the system before a failure occurs, reducing the impact of failures on system operations and users, and improving system availability and stability.

[0084] Specifically, in step S20, the stack nodes executed by the application server cluster nodes corresponding to the original log data and the corresponding network fault log data are normalized to obtain the normalized original log data, including the following steps:

[0085] Step S21: Normalize the stack nodes executed by the application server cluster nodes corresponding to the original log data to obtain normalized log data (i.e., the normalized original log data mentioned above); normalize the network fault log data corresponding to the original log data to obtain normalized log data; the normalized log data includes the set of normalized stack node original log data and the set of normalized network fault log data.

[0086] The final normalized stack node raw log data set is obtained as follows: Wx*={w1 * ,w2 * ,…wi* ,…wN *}, where N represents the total number of factors in the normalized stack node raw log data set, and wi * is the normalized stack node raw log data of the i-th stack node; the network fault log data set is obtained as follows: Wy*={w1 * ,w2 *,…wi * ,…wN *}, where N represents the total number of factors in the normalized network fault raw log data set, and wi * is the normalized network fault raw log data of the i-th network fault.

[0087] It should be noted that the technical solution of this application first calls multiple typical fault samples (i.e., samples containing abnormal log data of typical faults) from historical data; this application calls the above m samples (i.e., typical fault log samples), the above m samples contain corresponding k features, and we hope to filter abnormal log data based on factors such as the m samples, the j-th network fault, and the n-th abnormal stack fault. First, we can construct an m×k matrix A (i.e., the target matrix) to represent the log data after matrix operation processing, where each row represents a sample and each column represents a feature; the specific technical solution is as follows:

[0088] Specifically, such as Figure 3 As shown, in step S30, the normalized raw log data is analyzed and identified to filter out abnormal log data, including the following steps:

[0089] Step S31: During the initialization process, the normalized raw log data is analyzed and identified for the first time to obtain the first classification result; abnormal log data is then selected from the first classification result.

[0090] Step S32: Verify the first classification result to obtain the verification result; determine whether the verification result meets the target expansion condition. If so, start the weight factor introduction operation.

[0091] Step S33: Combine the weighting factors to perform a second analysis and identification on the normalized original log data to obtain a second classification result; filter out abnormal log data from the second classification result.

[0092] It should be noted that the normalized log data is first classified to obtain the first classification result. By using an appropriate model or algorithm, the log data can be quickly and accurately classified, and abnormal log data can be filtered out. This abnormal log data may contain information about system failures. The first classification result is then verified to obtain the verification result and determine whether the target expansion conditions are met. If the verification result does not meet the expected conditions, the weighting factor introduction operation is not performed; otherwise, the following operations are performed. Finally, the normalized original log data is classified a second time using the weighting factor to obtain the second classification result.

[0093] Weighting factors are used to model and analyze the correlations between different attributes or features in log data, thereby enabling a more comprehensive classification and identification of log data. The secondary classification results can more accurately filter out abnormal log data, improving the accuracy and efficiency of fault identification. By introducing weighting factors, the attribute correlations in log data can be considered more comprehensively, improving the accuracy of fault diagnosis. This method helps system administrators to promptly identify and resolve potential faults, improving the reliability and stability of cloud data computing systems. This method can efficiently classify and identify log data from cloud data computing systems, thereby predicting and diagnosing system faults.

[0094] Specifically, such as Figure 4 As shown, in step S31, during the initialization process, the normalized raw log data is analyzed and identified for the first time to obtain the first classification result, including the following steps:

[0095] Step S311: Obtain historical data, select m typical fault log samples from the historical data to construct a typical fault log sample set M={m1, m2, m3, ... mi}; where mi represents the i-th typical fault log sample; and obtain j feature vectors corresponding to each typical fault log sample to construct a feature vector set K={k1, k2, k3, ... kj}; where kj is the j-th feature vector; these typical fault log samples should represent the situation of typical faults containing abnormal log data.

[0096] Call the servers of multiple cloud data computing systems that have failed, obtain historical data (i.e. all log samples in history, which contain normal samples and also original log samples of typical failures) from the servers of the multiple cloud data computing systems, and filter multiple typical failure log samples from the historical data.

[0097] Step S312: Construct the target matrix A based on all the typical fault log sample sets and all the feature vector sets K:

[0098] Where, mikj is the j-th feature vector of the i-th typical fault sample log;

[0099] Step S313: Set a network fault weight vector W={w1,w2,wj...,wt} for each of the aforementioned feature vectors; where wj is the weight of the j-th feature vector;

[0100] It should be noted that the above network fault weight vector represents the first-level associated features: assuming the j-th network fault is related to the first t features, a network fault weight vector W of size k is constructed. W = {w1, w2, wt, ..., wk}; where wt represents the weight of the t-th feature. If the feature is unrelated to the network fault, its corresponding weight is 0.

[0101] Step S314: Set a stack fault weight vector N={n1,n2,...nj,ng} for each of the aforementioned feature vectors; where nj is the weight of the j-th feature in the fault stack;

[0102] It should be noted that the stack fault weight vector mentioned above represents the second-level associated features: assuming the nth abnormal stack is related to the following q features, a stack fault weight vector N of size k is constructed. N = {n1, n2, ..., nq, nk}; where nq represents the weight of the qth feature in the abnormal stack. If the feature is unrelated to the abnormal stack, its corresponding weight is 0.

[0103] Step S315: Calculate and obtain the fault feature vector F based on the network fault weight vector W and the stack fault weight vector N;

[0104] The fault feature vector F is calculated as follows:

[0105] F = W + N;

[0106] Where W+N represents the sum of corresponding elements;

[0107] It should be noted that the above embodiment of this application obtains the fault feature vector by adding the network fault weight vector w and the stack fault weight vector n element by element.

[0108] Fault feature vector F = W + N;

[0109] Where W+N represents the sum of corresponding elements to obtain the fault feature vector F.

[0110] Step S316: Calculate the comprehensive fault classification vector Y corresponding to each of the typical fault log samples based on the target matrix A and the fault feature vector F; input the comprehensive fault classification vector Y into the pre-trained classifier and output the first classification result;

[0111] The comprehensive fault classification vector Y is calculated as follows:

[0112] Y = F × A = {y1, y2, y3, ... yi, ... ym}; where yi is the comprehensive fault classification vector of the i-th typical fault log sample;

[0113] The first classification result is used to indicate whether the current i-th sample is at risk;

[0114] It should be noted that, finally, we can calculate the product between matrix A and the fault feature vector f, obtaining a vector Y = {y1, y2, y3, ..., yi, ..., ym} of length m, where yi represents the degree of correlation between the i-th sample and the (comprehensive fault). Inputting the value of yi into the classifier yields the first classification result. Based on this result, we can filter out fault log data for further fault classification and identification. The first classification result indicates whether the current i-th sample carries a risk of (comprehensive fault).

[0115] By analyzing historical data from multiple cloud computing systems, typical fault log samples were extracted, and corresponding feature vectors and target matrices were constructed based on these samples. Simultaneously, by setting network fault weight vectors and stack fault weight vectors, the features related to network and stack faults in the fault log samples can be more accurately reflected. Finally, by calculating the product of the fault feature vector and the target matrix, a comprehensive fault classification vector is obtained, which can then determine whether the current sample carries a fault risk.

[0116] This method improves the accuracy and efficiency of fault diagnosis, helping to quickly identify and resolve system faults and enhance system reliability and stability. Furthermore, it can classify fault samples, helping users assess fault risks and take timely and appropriate measures.

[0117] See Figure 1The method used in this application embodiment combines the ELK platform 100, which is a centralized log processing solution. It uses open-source tools such as ElasticSearch module 101, Logstash module 102, Elasticsearch cluster module 103, and Kiabana module 104 to meet the query, sorting, and statistical needs of large-scale cluster logs.

[0118] The ELK platform 100 (i.e., the main controller) centrally manages the logs from the collection module (ElasticSearch module 101) to the Logstash module 102. The logs are then formatted (Logstash module 102) and normalized before being output to the Elasticsearch cluster module 103. Next, the log data is indexed to its storage path and its location is locked to the ElasticSearch module 101, and recorded in the cluster node log data graph. In essence, the Logstash module collects the raw log data generated by the collection module (ElasticSearch module 101), stores it in the Elasticsearch-managed cluster nodes, and performs formatting (Logstash module 102) and normalization operations. Finally, after identifying and locking the target fault, the storage location of the faulty log data can be located by querying the cluster node log data graph, and the log file at the storage location can be directly accessed for further analysis.

[0119] However, Kiabana module 104 ultimately queries data from the Elasticsearch-managed cluster nodes to generate cluster node log data charts after identifying the target fault, and then returns them to the main controller (which then directly calls the log files in the storage location for further analysis).

[0120] Specifically, such as Figure 5 As shown, in step S32, it is determined whether the verification result meets the target expansion condition. If so, the weight factor introduction operation is initiated, including the following steps:

[0121] Step S321: Pre-set secondary evaluation thresholds; the secondary evaluation thresholds include error rate standard thresholds and stack fault evaluation standard thresholds;

[0122] Step S322: Verify the first classification result of the cloud data computing system using the secondary evaluation threshold to obtain the verification result;

[0123] It should be noted that the verification method for verifying the classification results of the cloud data computing system using the secondary evaluation threshold includes: calculating the overall classification error rate based on the first classification result; determining whether the overall classification error rate in the current first classification result of a preset number of original log data is higher than the error rate standard threshold; if it is higher than the error rate standard threshold, then further filtering multiple abnormal log data that actually have stack faults.

[0124] The number of first-class results of abnormal log data that incorrectly predict stack faults is counted. The stack fault judgment error rate is obtained by dividing the number of first-class results of abnormal log data that incorrectly predict stack faults by the proportion of the total number of samples of abnormal log data that actually have stack faults. Then, it is determined whether the stack fault judgment error rate is greater than the stack fault evaluation standard threshold.

[0125] Step S323: Determine whether the verification result meets the preset verification error rate condition. If so, start the weight factor introduction operation.

[0126] The verification error rate condition refers to: the overall classification error rate is higher than the error rate standard threshold and the stack fault judgment error rate is greater than the stack fault evaluation standard threshold.

[0127] The aforementioned target extension conditions refer to the secondary evaluation thresholds, namely the error rate standard threshold and the stack fault evaluation standard threshold. By comprehensively judging the overall classification error rate and the stack fault judgment error rate, if the overall classification error rate is higher than the error rate standard threshold in the current first classification result of a preset number of raw log data, and at the same time it is judged whether the stack fault judgment error rate is greater than the stack fault evaluation standard threshold, the weight factor operation is executed to improve the accuracy of fault prediction.

[0128] It should be noted that, in a further technical solution of the above-described embodiment of this application, the first classification result is verified to obtain a verification result; it is determined whether the verification result meets the preset target expansion conditions. If so, the weight factor introduction operation is initiated, and the weight factor introduction operation is controlled to classify the sample; if not, the above step S40 is directly performed to identify the abnormal stack information of the abnormal log data to determine the source of the fault.

[0129] In the above embodiments of this application, since the fault prediction capability is seriously inconsistent with the actual situation, the first classification result obtained may be due to abnormal load caused by access volume and other reasons, which may further lead to misjudgment of stack fault or network fault, resulting in a large error in the classification result. Therefore, it is necessary to introduce a weight factor, i.e., a third-level weight factor, to obtain a more accurate second classification result, thereby ultimately improving the accuracy of fault prediction.

[0130] Preferably, as an alternative implementation, after introducing the weighting factor operation, the correction and identification processing operation for the abnormal log data should still be repeated, as detailed in the following processing procedure.

[0131] It should be noted that research has found that when detecting specific fault scenarios, if the model performs poorly in these scenarios, it may be due to a third, crucial influencing factor causing frequent faults that were not considered. For example, under high traffic conditions, the fault rate is high, but there is a misjudgment problem; load faults are misjudged as stack faults. Therefore, high traffic may be a third factor triggering load faults in this situation, and a significant factor leading to misjudgments of stack faults. Thus, this invention provides an in-depth understanding and analysis of specific fault scenarios, not only identifying shallow fault scenarios but also further discovering other fault scenarios that may lead to misjudgments of stack faults. Classifying and analyzing these scenarios, determining whether to introduce a third-level weight vector is a crucial in-depth discovery process.

[0132] Specifically, such as Figure 6 As shown, in S33, the normalized original log data is analyzed and identified a second time using the weighting factors to obtain a second classification result, specifically including:

[0133] Step S331: Obtain the e extended feature vectors K'={k'1, k'2, k'3, ...k'e} corresponding to the typical fault log sample; where k'e is the e-th extended feature vector; construct the extended target matrix A' based on the extended feature K' and the typical fault log sample set M;

[0134] The extended target matrix A' is represented as follows:

[0135] Where, mik'e is the e-th extended feature vector of the log of the i-th typical fault sample;

[0136] It should be noted that the above embodiment of this application first constructs a target matrix: the e extended features contained in these i samples are combined into an m×k' matrix A' (i.e., the extended target matrix), where each row represents a sample and each column represents a feature; that is, the extended feature parameters of the number of visits are reintroduced into it;

[0137] Step S332: For each of the extended feature vectors, preset an extended network fault weight vector W'={w'1,w'2,w'j...,w't}; where w'j is the weight of the j-th extended feature vector in the network fault;

[0138] It should be noted that the above extended network fault weight vector is the first-level associated feature. Assuming that the j-th network fault is related to the first t features, a network fault weight vector W' of size t is constructed. If the extended feature is not related to the network fault, the corresponding weight is 0.

[0139] Step S333: Construct an extended stack fault weight vector N'={n'1,n'2,...n'j,n'g} for each of the extended feature vectors; where n'j is the weight of the j-th extended feature vector in the fault stack;

[0140] It should be noted that the above extended stack fault weight vector is the second-level associated feature. Assuming that the j-th abnormal stack is related to the next g features, a stack fault weight vector N' of size g is constructed. If the feature is not related to the abnormal stack, the corresponding weight is 0.

[0141] Step S334: Construct a third-level fault weight vector P={p1,p2,..pj,pf} for each of the extended feature vectors; where pf is the weight of the f-th extended feature vector in the third-level fault.

[0142] It should be noted that the third-level fault weight vector mentioned above is the access volume weight vector. Assuming that the j-th access volume is related to the following f features, an access volume weight vector P of size f is constructed. If the extended feature is not related to the access volume, the corresponding weight is 0.

[0143] Step S335: Calculate and obtain the extended fault feature vector F' based on the extended network fault weight vector W', the extended stack fault weight vector N', and the third-level fault weight vector P (or the weight of the access volume weight vector mentioned above).

[0144] The extended fault feature vector F' is calculated as follows:

[0145] F' = W' + N' + P;

[0146] Where W'+N'+P is the sum of corresponding elements;

[0147] It should be noted that the above-described embodiment of this application obtains the extended fault feature vector by element-wise summing of the extended network fault weight vector W', the extended stack fault weight vector N', and the third-level fault weight vector P.

[0148] Step S336: Calculate and obtain the extended comprehensive fault classification vector Y' corresponding to each of the typical fault log samples based on the extended target matrix A' and the extended fault vector F'; input the extended comprehensive fault classification vector Y' into the pre-trained classifier and output the second classification result;

[0149] The extended comprehensive fault classification vector Y' is calculated as follows:

[0150] Y'=F'×A'={y'1,y'2,y'3,...y'i,...y'm};where y'i is the comprehensive fault classification vector of the i-th typical fault log sample;

[0151] The second classification result is used to indicate whether the current i-th sample is at risk;

[0152] It should be noted that in the above technical solution, when considering the third-level failure factor, the introduction of the weight factor P can further improve the performance of the model; the introduction of the weight factor P leads to the expansion of the three-dimensional weight parameters; the role of the weight factor P is as follows: the third-level weight P is a vector of size k, representing the weight of the third level; the introduction of P is to consider the new failure factors that may emerge in the wide development of sample data, making the model more flexible.

[0153] The weight factor P in the above embodiments of this application can be determined by means of expert knowledge, feature importance analysis, etc.; if there is new abnormal log data, P can be updated by supervised learning or semi-supervised learning; the above weight factor is P={p1,p2,..pj,pf}; after the weight factor is introduced: the target matrix expansion processing operation needs to be performed: construct the expanded target matrix A', the fault feature vector F' which includes network fault weight, stack fault weight and the introduced third-level weight (such as the influence weight of access volume on the occurrence of fault logs), and then perform the product calculation of the feature vector and the expanded target matrix to obtain the expanded comprehensive fault classification vector Y', so as to ensure that in subsequent operations, multiple dimensions of factors causing the generation of fault logs can be considered, thereby more accurately identifying and analyzing fault logs, and more quickly and accurately identifying fault locations for maintenance processing.

[0154] Regarding the issues related to log data processing, researchers have found that the main problems with log data are as follows: First, the information in the log data is unclear, resulting in reduced execution efficiency (in this embodiment, the corresponding location of the application server cluster nodes is clear); second, the format of the log data is not standardized, resulting in difficulty in conveniently reading key information and hindering convenient data collection (the method used in this embodiment is to normalize the execution stack nodes of the corresponding application server cluster nodes and the corresponding network fault log data); third, the log data is scarce and lacks key information (i.e., it lacks key features that affect the occurrence of faults), resulting in reduced efficiency in locating problems (the method used in this embodiment is to continuously update, iterate, dynamically adjust, and promptly discover new key features that affect the occurrence of faults, thereby smoothly introducing third-party weighting factors to solve this problem).

[0155] Specifically, such as Figure 7 As shown, in step S334, a third-level fault weight vector P={p1,p2,..pj,pf} is constructed for each of the extended feature vectors, including the following steps:

[0156] Step S3341: Obtain the third-level correlation features (generally access volume data) corresponding to each of the typical fault log samples.

[0157] Step S3342: Normalize the third-level association features to obtain normalized association feature data;

[0158] Step S3343: Statistically analyze the normalized correlation feature data to obtain the distribution information corresponding to the third-level correlation feature;

[0159] Step S3344: Assign weights to each of the typical fault log samples based on the distribution information to obtain the third-level fault weight vector corresponding to each of the extended feature vectors;

[0160] It should be noted that the above embodiments of this application first extract third-level correlation features from typical fault log samples, such as access volume data. Third-level correlation features are usually related to the occurrence and impact of faults. By collecting these features, the contextual information of fault events can be described more comprehensively. Furthermore, normalization processing helps to transform third-level correlation features of different ranges and units into the same scale, so that different features can be better compared and processed in subsequent statistical and weight calculation processes. By statistically analyzing the distribution information of third-level correlation features, it is helpful to understand the overall distribution of these features, such as statistical measures such as mean and variance. This information can provide a basis for subsequent weight calculation, making the weights more consistent with the actual situation. By assigning weights to each typical fault log sample based on distribution information, the importance of each sample in constructing the fault weight vector can be more reasonably reflected. For constructing extended feature vectors, these weights can be used for weighted averaging to emphasize the influence of certain key features.

[0161] By using normalization and statistical distribution information, more comprehensive and comparable feature information is provided for subsequent weight calculation. The weights are allocated according to the relative position of the samples in the overall distribution, so that more representative or influential samples can have a greater impact on the final weights when constructing the fault weight vector. Through the above scheme, the fault weight vector can be more adapted to the actual situation, better reflect the influence of each sample in the whole system, and improve the model's ability to predict and diagnose actual faults.

[0162] This application's embodiments employ a technical solution that analyzes log files and text features. Note that this analysis process has feedback and iterative capabilities, allowing the identification of log data recording exception stacks and the extraction of exception information. These exception stacks typically contain the exception type, error cause, location on the call stack, and related information. Developers can use this exception stack information to trace the source of the problem and perform appropriate debugging and repairs.

[0163] In summary, the cloud platform cluster network security log data fault prediction and processing method proposed in this invention first obtains raw log data containing various records and information during system operation, then performs normalization processing on the raw log data; further, it performs row analysis and identification on the normalized raw log data to filter out abnormal log data; finally, it performs further analysis on the abnormal log data, especially identifying and determining the source of the fault based on the abnormal stack information.

[0164] In the specific process of filtering out abnormal log data, the normalized log data is first classified to obtain the first classification result. The first classification result is then verified to obtain the verification result and to determine if it meets the target expansion conditions. If the verification result does not meet the expected conditions, no weighting factor is introduced, and abnormal log data is directly filtered from the first classification result. If the verification result meets the expected conditions, a weighting factor is introduced, and finally, the normalized original log data is classified a second time using the weighting factor to obtain the second classification result. The weighting factor can more comprehensively classify and identify log data, thus the second classification result can more accurately filter out abnormal log data, improving the accuracy and efficiency of fault identification. This method can efficiently classify and identify log data from cloud data computing systems, thereby predicting and diagnosing system faults.

[0165] In the initial classification process, historical data from multiple cloud computing systems is analyzed to extract typical fault log samples, and corresponding feature vectors and target matrices are constructed based on these samples. Simultaneously, by setting network fault weight vectors and stack fault weight vectors, the features related to network and stack faults in the fault log samples can be more accurately reflected. Finally, by calculating the product of the fault feature vector and the target matrix, a comprehensive fault classification vector is obtained, which can then determine whether the current sample poses a fault risk.

[0166] This method improves the accuracy and efficiency of fault diagnosis, helping to quickly identify and resolve system faults and enhance system reliability and stability. Furthermore, it can classify fault samples, helping users assess fault risks and take timely and appropriate measures.

[0167] Meanwhile, in the process of determining whether to introduce a weighting factor, since the fault prediction capability is seriously inconsistent with reality, the first classification result may be abnormal due to factors such as access volume, which may further lead to the identification of stack faults or network faults, resulting in a large error in the classification result. Therefore, it is necessary to introduce a weighting factor to obtain a more accurate second classification result, thereby ultimately improving the accuracy of fault prediction.

[0168] In the specific analysis to obtain the second classification result, when considering the third-level fault factor, introducing the weight factor P can further improve the model performance.

[0169] The introduction of the weight factor P leads to the expansion of the weight parameters in three dimensions; the role of the aforementioned weight factor P is as follows: the third-level weight vector P is a vector of size k, representing the weights of the third level; the introduction of P is to take into account the new failure factors that may emerge in the wide development of sample data, making the model more flexible.

[0170] Furthermore, in constructing the weight factor P, third-level correlation features, such as access volume data, are first extracted from typical fault log samples. These third-level correlation features are usually related to the occurrence and impact of faults. By collecting these features, the contextual information of fault events can be described more comprehensively. Normalization helps to transform third-level correlation features of different ranges and units into the same scale, so that different features can be better compared and processed in subsequent statistical and weight calculations. Statistical analysis of the distribution information of third-level correlation features helps to understand the overall distribution of these features, such as the mean and variance. This information provides a basis for subsequent weight calculations, making the weights more consistent with reality. By assigning weights to each typical fault log sample based on distribution information, the importance of each sample in constructing the fault weight vector can be more reasonably reflected. For constructing extended feature vectors, these weights can be used for weighted averaging, emphasizing the influence of certain key features.

[0171] By using normalization and statistical distribution information, more comprehensive and comparable feature information is provided for subsequent weight calculation. The weights are allocated according to the relative position of the samples in the overall distribution, so that more representative or influential samples can have a greater impact on the final weights when constructing the fault weight vector. Through the above scheme, the fault weight vector can be more adapted to the actual situation, better reflect the influence of each sample in the whole system, and improve the model's ability to predict and diagnose actual faults.

[0172] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; those skilled in the art can modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cloud platform cluster network security log data failure prediction processing method, characterized in that, The method comprises the following steps: obtain raw log data of a cloud data computing system; normalize the stack nodes executed by the application server cluster nodes corresponding to the raw log data and the network fault log data corresponding to the raw log data respectively to obtain normalized raw log data; analyze and identify the normalized raw log data to filter out abnormal log data, which comprises the following steps: during initialization processing, first analyze and identify the normalized raw log data to obtain a first classification result; filter out abnormal log data from the first classification result; verify the first classification result to obtain a verification result; determine whether the verification result meets a target expansion condition; if yes, start a weight factor introduction operation; second analyze and identify the normalized raw log data in combination with the weight factor to obtain a second classification result; filter out abnormal log data from the second classification result; during initialization processing, first analyze and identify the normalized raw log data to obtain a first classification result, which comprises the following steps: obtain historical data, select m typical fault log samples from the historical data to construct a typical fault log sample set M={m1, m2, m3,... mi}; wherein mi represents the i-th typical fault log sample; and obtain j feature vectors corresponding to each of the typical fault log samples to construct a feature vector set K={k1, k2, k3,... kj}; wherein kj is the j-th feature vector; construct a target matrix A based on all the typical fault log sample sets and all the feature vector sets K: ; where mikj is the jth feature vector of the ith typical failure sample log; set a network fault weight vector W={w1, w2, wj,..., wt} for each of the feature vectors; wherein wj is the weight of the j-th feature vector; set a stack fault weight vector N={n1, n2,..., nj, ng} for each of the feature vectors; wherein nj is the weight of the j-th feature in the abnormal stack; calculate a fault feature vector F based on the network fault weight vector W and the stack fault weight vector N; the calculation method of the fault feature vector F is F=W+N; wherein W+N is the sum of corresponding elements; calculate a comprehensive fault classification vector Y corresponding to each of the typical fault log samples according to the target matrix A and the fault feature vector F; input the comprehensive fault classification vector Y into a pre-trained classifier to output a first classification result; the calculation method of the comprehensive fault classification vector Y is Y=F×A={y1, y2, y3,... yi,... ym}; wherein yi is the comprehensive fault classification vector of the i-th typical fault log sample; the first classification result is used to indicate whether the current i-th sample is at risk; identify abnormal stack information of the abnormal log data to determine the fault source. 2.The cloud platform cluster network security log data fault prediction processing method of claim 1, wherein, The normalized raw log data comprises a normalized stack node raw log data set and a normalized network fault log data set. 3.The cloud platform cluster network security log data fault prediction processing method of claim 2, wherein, The normalization processing of the stack nodes executed by the application server cluster nodes corresponding to the original log data and the network fault log data corresponding to the original log data is performed to obtain normalized original log data, and the normalized original log data includes the following operation steps: The stack nodes executed by the application server cluster nodes corresponding to the original log data are normalized to obtain normalized log data, and the network fault log data corresponding to the original log data is normalized to obtain normalized log data.

4. The cloud platform cluster network security log data fault prediction processing method of claim 3, wherein, The verification result is judged to determine whether the target expansion condition is reached, and if so, the weight factor introduction operation is started, including the following operation steps: A secondary evaluation threshold is set in advance; the first classification result of the cloud data computing system is verified by using the secondary evaluation threshold to obtain a verification result; It is judged whether the verification result reaches the preset verification error rate condition, and if so, the weight factor introduction operation is started. The verification error rate condition refers to that the overall classification error rate is higher than the error rate standard threshold and the stack fault judgment error rate is greater than the stack fault evaluation standard threshold. 5.The cloud platform cluster network security log data fault prediction processing method of claim 4, wherein, The secondary evaluation threshold includes the error rate standard threshold and the stack fault evaluation standard threshold.

6. The cloud platform cluster network security log data fault prediction processing method of claim 5, wherein, The first classification result of the cloud data computing system is verified by using the secondary evaluation threshold to obtain a verification result, including the following operation steps: The overall classification error rate is calculated according to the first classification result; it is judged whether the overall classification error rate is higher than the error rate standard threshold, and if so, the multiple abnormal log data actually existing stack faults are further selected; The number of abnormal log data actually existing stack faults is counted, and the proportion of the sample number of the first classification result of the predicted error abnormal log data in the abnormal log data actually existing stack faults is calculated. It is judged whether the stack fault judgment error rate is greater than the stack fault evaluation standard threshold.

7. The cloud platform cluster network security log data fault prediction processing method of claim 6, wherein, The normalized original log data is analyzed and identified for the second time by combining the weight factor to obtain a second classification result, specifically including: The e extended feature vectors K'={k'1, k'2, k'3,...k'e} corresponding to the typical fault log sample are obtained; wherein k'e is the e extended feature vector; the extended target matrix A' is constructed based on the extended feature K' and the typical fault log sample set M; The extended target matrix A' is expressed as: ; where mik'e is the e-th extended feature vector of the i-th typical failure sample log; The extended network fault weight vector W'={w'1, w'2, w'j..., w't} is preset for each extended feature vector; wherein w'j is the weight of the jth extended feature vector in the network fault; The extended stack fault weight vector N'={n'1, n'2,...n'j, n'g} is constructed for each extended feature vector; wherein n'j is the weight of the jth extended feature vector in the abnormal stack; The third-level fault weight vector P={p1, p2,..pj, pf} is constructed for each extended feature vector; wherein pf is the weight of the fth extended feature vector in the third-level fault; Based on the extended network fault weight vector W' and the extended stack fault weight vector N', the third-level fault weight vector P is calculated to obtain an extended fault feature vector F'; The calculation method of the extended fault feature vector F' is: F'=W'+N'+P; Wherein, W'+N'+P is the addition of corresponding elements; According to the extended target matrix A' and the extended fault vector F', an extended comprehensive fault classification vector Y' corresponding to each typical fault log sample is calculated; the extended comprehensive fault classification vector Y' is input into a pre-trained classifier to output a second classification result; The calculation method of the extended comprehensive fault classification vector Y' is: Y'=F'×A'={y'1,y'2,y'3,...y'i,...y'm}; wherein, y'i is the comprehensive fault classification vector of the i th typical fault log sample; The second classification result is used to indicate whether the current i th sample exists risk. 8.The cloud platform cluster network security log data fault prediction processing method of claim 7, wherein, The third-level fault weight vector P={p1,p2,..pj,pf} corresponding to each extended feature vector is constructed, including the following operation steps: Obtaining the third-level associated features corresponding to each typical fault log sample; The third-level associated features are normalized to obtain normalized associated feature data; The distribution information corresponding to the third-level associated features is obtained by statistical processing on the normalized associated feature data; Based on the distribution information, weights are assigned to each typical fault log sample to obtain the third-level fault weight vector corresponding to each extended feature vector.

Images