Computer fault early warning system based on big data

Abstract

The invention discloses a computer fault early warning system based on big data, which relates to the technical field of computer system monitoring and comprises a big data network module, a computer data terminal, a fault analysis module, a fault processing module and a fault early warning module. Multi-source fault data is collected through the big data network module, a fault scene is generated by using an association rule, and early recognition of a complex fault is realized in combination with time and equipment association. The computer data terminal collects multi-dimensional operation data, detects abnormity through statistical analysis, clustering and machine learning, and outputs feature vectors. The fault analysis module matches the abnormal features with the fault scene to generate an initial fault scene, and the fault processing module predicts an offset path through multi-directional evolution and a long-short-term memory network and a Prophet model and calculates offset and complexity. And the fault early warning module is combined with task scheduling to perform conflict detection and dynamically trigger early warning, so that the system reliability is improved and the resource use is optimized.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of computer system monitoring, and particularly relates to a computer fault early warning system based on big data. BACKGROUND

[0002] With the wide application of computers in enterprise management, financial transactions, scientific research computing, intelligent manufacturing and other fields, the operation stability and data security of computers have become increasingly important. Various hardware faults or software faults may occur in computer systems during long-term operation, which often seriously affects business continuity and data integrity.

[0003] However, in the prior art, computer fault monitoring and early warning mainly rely on manual inspection, single index threshold alarm or simple log analysis systems, which have the following main problems: traditional systems usually rely on a single type of data and cannot comprehensively reflect the system operation state, making it difficult to capture the signs of complex faults in a timely manner; the traditional early warning method simply alarms after discovering the abnormality, without dynamic adjustment in combination with the current task scheduling and computing resource usage, which is easy to cause system performance waste or business interruption; the prior art cannot simulate the multi-directional evolution of the initial fault scene, lacks prediction and conflict detection of the offset fault path, and the early warning strategy is static and has a lagging reaction. SUMMARY

[0004] The application aims to solve the problems mentioned in the background and proposes a computer fault early warning system based on big data.

[0005] The application can be achieved by the following technical scheme: a computer fault early warning system based on big data, comprising: a big data network module, a computer data terminal, a fault analysis module, a fault handling module and a fault early warning module. The big data network module stores various computer data, and constructs a fault scene set according to various computer data; The computer data terminal is used for collecting running data of the data for abnormal analysis to obtain an abnormal feature vector set; The fault analysis module obtains the abnormal feature vector set and fault scene records for diagnosis; the fault scene record set is diagnosed by using a two-stage algorithm model to obtain a candidate scene; the obtained candidate scene is input into a gradient boosting decision tree (GBDT) model to output an initial fault scene set; The fault handling module simulates the development of each initial fault scene in the obtained initial fault scene set to obtain multi-directional offset fault scenes; The fault early warning module obtains an offset fault scene path according to the multi-direction offset fault scene, and calculates an offset amount of each offset fault scene; the complexity of the fault is calculated according to the offset amount, and a pre-processing time length of the fault is obtained according to the complexity of the fault; after the pre-processing time length of the fault, a current computer task path set being executed is obtained, a conflict detection is performed between the processing path of the current offset fault scene and the task path set, and a conflict state value is obtained.

[0006] As a preferred embodiment of the present application, the specific process of constructing a fault scene according to various types of computer data is as follows: The computer fault type data is obtained from the Internet through a network crawler or an interface call, and corresponding fault data is generated by using an association rule analysis algorithm; the derived fault data set is combined and associated to form various fault scenes.

[0007] As a preferred embodiment of the present application, the process of generating corresponding fault data by using an association rule analysis algorithm is as follows: The obtained fault type data is cleaned and standardized to construct a transaction set; based on the transaction set, a minimum support threshold is set, and the Apriori algorithm is used to calculate the transaction set to obtain the occurrence frequency of each fault type and its combination in the transaction set; the fault type combination with a support greater than or equal to the minimum support threshold is selected according to the historical data distribution to dynamically adjust the minimum support threshold, a frequent item set is obtained, and the frequent item set is sorted according to the size of the support; a candidate rule is generated for each frequent item set, the confidence of each candidate rule is calculated, and the rules with a confidence greater than or equal to a minimum confidence threshold are selected as effective association rules according to the confidence of the candidate rules; based on the obtained effective association rules, the fault type events corresponding to the rules are mapped to the corresponding fault data attributes, and a fault data set is obtained by statistics.

[0008] As a preferred embodiment of the present application, the specific process of combining and associating the derived fault data set is as follows: Matching is performed based on the temporal and device correlation of fault data: the temporal correlation is determined by comparing the occurrence time of each fault data in the fault data set and calculating the time difference between any two fault data; when the time difference is less than a preset time window, any two fault data are determined to be correlated in time; the device correlation is determined by cross-comparing the sets of devices involved in the fault data: the sets of devices involved in two fault data are compared one by one, the number of device pairs with topological relationships is counted, and the device correlation index is calculated based on the size of the combined device sets; the calculated device correlation index is compared with a set device correlation threshold, and if the device correlation index is greater than or equal to the device correlation threshold, it is determined that the two fault data are correlated on the device, and a corresponding fault scenario record is generated.

[0009] In a preferred embodiment of the present invention, the process of using a computer data terminal to collect operational data and perform anomaly analysis is as follows: The computer data terminal collects operational data in real time according to the set sampling frequency; the raw operational data is preprocessed to obtain operational data; the operational data is mapped to a multidimensional feature space to obtain multidimensional features; the multidimensional features are then compared with the historical normal operation dataset stored in the big data network module to obtain baseline features; and features that exceed the threshold of a single or combined baseline feature are initially marked as anomalies. The designed multi-model parallel detection mechanism includes: using statistical methods to detect anomalies in the running data to obtain running numerical distribution anomalies; then using cluster analysis to identify the offset of the running state cluster centers, denoted as the running state offset; and combining the obtained running numerical distribution anomalies and the running state offset with a machine learning model to identify potential anomalies in complex patterns, specifically: The abnormal distribution of running numerical values ​​and the offset of running state are standardized. The standardized results are then fused with the feature vectors of the corresponding abnormal distribution of running numerical values ​​and the offset of running state in chronological order to obtain a multi-dimensional feature input matrix. The multi-dimensional feature matrix is ​​then input into a pre-trained machine learning model, such as an isolated forest model or a support vector machine anomaly detection model. During runtime, the machine learning model uses the feature boundaries of normal and abnormal patterns learned during the training phase to perform pattern matching and deviation calculation on the input data to obtain the feature deviation. When the feature deviation exceeds the feature deviation threshold set by the machine learning model, the feature sample is determined to be a potential anomaly, and an anomaly feature label and corresponding confidence score are output as anomaly data features in complex patterns. A set of anomaly feature vectors is then statistically obtained.

[0010] As a preferred embodiment of the present invention, the fault scenario record set is used for diagnosis using a two-stage algorithm model: The FAISS vector retrieval algorithm is used to construct a fault scene vector index. For each abnormal feature vector, the top-K nearest neighbor fault scene vectors are retrieved from the index. The similarity is measured using cosine similarity. The similarity is compared with the similarity threshold corresponding to the candidate scene. If the similarity is greater than the similarity threshold, it is considered a candidate scene. Based on the obtained candidate scenes, the features of the candidate scenes are concatenated with the abnormal feature vectors to form the input feature matrix, which is then input into the Gradient Boosting Tree (GBDT) model. The model has learned the mapping relationship between historical anomalies and fault scenes during the training phase. The fault type probability corresponding to each candidate scene is output according to the mapping relationship. The fault type with the highest probability is then selected as the initial fault scene set.

[0011] In a preferred embodiment of the present invention, the process of obtaining the offset fault scene path based on the multi-directional offset fault scene is as follows: Starting from the initial set of fault scenarios, a multi-directional evolution simulation algorithm is used to generate a set of offset fault scenarios for each initial scenario under various external perturbations. A multi-dimensional feature matrix is ​​constructed for each offset scenario and input into a long short-term memory network. The long short-term memory network updates the hidden state and unit state at each time step. Finally, the event type probability at each time step is obtained through the softmax output layer. The event type probability is compared with the event type probability threshold. If the event type probability is greater than the event type probability threshold, it is recorded as a high-confidence event. For trend indicators, the Prophet model is used to decompose the trend indicator into trend, seasonality and residual. Then, the event sequence predicted by the Long Short-Term Memory Network is time-aligned with the Prophet predicted indicator to generate the offset fault scene path. Based on the offset fault scene path, the dynamic time warping algorithm is used to align the offset path with the original scene path to obtain the offset amount.

[0012] In a preferred embodiment of the present invention, the specific process of calculating the complexity of the fault from the offset and determining the preprocessing time of the fault based on the complexity of the fault is as follows: Based on the calculated offset, a complexity vector is constructed, and the fault complexity is obtained through weighted linear-nonlinear fusion. The calculated complexity value is then input into the duration prediction model f. θ (·) Obtain the preprocessing time.

[0013] As a preferred embodiment of the present invention, the specific process of performing conflict detection between the processing path of the current offset fault scenario and the task path set is as follows: Obtain the set of tasks currently being executed by the computer, extract the corresponding execution paths, and form a task path set; perform conflict detection between the processing path of the current offset fault scenario and the task path set, specifically: For task paths and offset fault paths, if the time intervals overlap and the execution node sets intersect, a conflict is considered. The conflict state value is obtained through a formula. If the conflict state value is one for any task, the offset fault scenario is marked as a conflict offset fault scenario, and a fault warning is triggered. If there is no conflict, the process proceeds to the next round of offset fault scenario evolution, including: Based on the current scenario, a new batch of offset scenarios is obtained by simulating the development of each initial fault scenario to obtain multi-directional offset fault scenarios. Similarly, the conflict state value is recalculated for each new scenario. The iteration terminates when any of the following situations occur: all generated scenarios are in conflict and a warning has been triggered, or the scenario evolution reaches the set upper limit.

[0014] Compared with the prior art, the beneficial effects of the present invention are: 1. This invention collects multi-source computer fault data in real time through a big data network module, and uses association rule analysis to generate frequent itemsets and effective fault rules to construct a fault scenario set; by combining time correlation and equipment topology relationship, it performs multi-dimensional matching and mapping of fault events to achieve early identification of potential complex faults; compared with existing methods based on a single data source or static threshold, this invention improves the coverage and accuracy of fault prediction.

[0015] 2. This invention employs statistical analysis, cluster analysis, and machine learning models to perform real-time anomaly analysis on multi-dimensional operational data such as CPU utilization, memory usage, disk I / O, network throughput, and process status. It compares anomaly features with historical normal data baselines and generates a multi-dimensional feature matrix through feature fusion, which is then input into the model to identify potential anomalies under complex patterns. Compared with existing technologies, this invention captures single or combined feature anomalies in a short time, improving the response speed and recognition accuracy of anomaly detection, while reducing false positives and false negatives.

[0016] 3. This invention, based on an initial fault scenario, employs a multi-directional evolutionary simulation algorithm to generate offset fault scenarios. It then combines a Long Short-Term Memory (LSTM) network with a Prophet model to predict event sequences and trend indicators, forming an offset fault path. By calculating the offset, fault complexity, and preprocessing time, and combining this with conflict detection based on the current task scheduling path, the invention determines the risk of the offset scenario and triggers an early warning. Compared to existing static threshold alarm methods, this invention implements an intelligent and dynamic fault warning strategy, optimizes the efficiency of computing resource utilization, and rationally schedules task execution while detecting potential faults in advance, reducing the risk of business interruption. Attached Figure Description

[0017] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.

[0018] Figure 1 This is a schematic diagram of the principle of the present invention. Detailed Implementation

[0019] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0020] It should be understood that the terms “comprising” and “including” used in this disclosure and claims indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0021] It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this disclosure. As used in this disclosure and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this disclosure and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0022] Please see Figure 1 As shown, a computer fault early warning system based on big data includes: a big data network module, a computer data terminal, a fault analysis module, a fault processing module, and a fault early warning module; The big data network module stores various types of computer data and constructs a set of fault scenarios based on this data; specifically: Computer fault type data is obtained from the Internet through web crawlers or API calls; the fault types include, but are not limited to, hardware faults, software anomalies, network connection failures, and system performance anomalies; the obtained data includes the fault occurrence time, fault device type, fault description information, fault logs, and related operating parameters, and the data is cleaned, deduplicated, and standardized.

[0023] Based on the fault type data, the corresponding fault data is generated using an association rule analysis algorithm, specifically: The acquired fault type data is cleaned and standardized, and all fault types are categorized and summarized. Duplicate fault types are removed, and each fault transaction is then classified into its corresponding fault type to construct a transaction set, where the transaction set T = {t1, t2, ..., t...} i, ..., t n}, where each transaction t i For each fault event, the fault type and related attributes are included. Based on the transaction set T, a minimum support threshold T0 is set. The Apriori algorithm is used to scan the transaction set T to obtain the frequency of occurrence of each fault type and its combination in the transaction set. The minimum support threshold T0, with a value range of [0.01, 0.1] and a default of 0.05, is dynamically adjusted according to the historical data distribution. Fault type combinations with support greater than or equal to the minimum support threshold T0 are selected to obtain the frequent itemset F = {f1, f2, ..., f...}. j , ..., f m}, where each f j It contains one or more associated fault types; the frequent itemsets F are sorted according to their support.

[0024] For each frequent itemset f j Candidate rules are generated in the following form: ,in Y=f j \X; where X and Y are sets of fault types; then calculate the confidence level of each candidate rule. Through the formula: Where Support(X∪Y) represents the proportion of transactions in the entire set of failed transactions that simultaneously contain both failure type sets X and Y, i.e., the frequency of rule occurrence; Support(X) represents the proportion of transactions containing failure type set X, i.e., the frequency of rule conditions occurring; based on the confidence level of candidate rules... Filter out those with a confidence level greater than or equal to the minimum confidence threshold The rules are used as valid association rules; the minimum confidence threshold The value range is [0.5, 0.9], with a default value of 0.7.

[0025] Based on the obtained effective association rules, the fault type events corresponding to X on the left side of the rule are mapped to the fault data attributes corresponding to Y on the right side, and statistics are performed to obtain the fault data set D={d1, d2, ..., d... o , ..., d k}

[0026] The derived fault data sets are combined and correlated to form various fault scenarios, specifically: Matching is performed based on the temporal and device correlations between fault data: The time correlation is determined by comparing the occurrence time of each fault data in the fault data set and calculating the time difference Δt between any two fault data; when the time difference is less than the preset time window ΔT, it is determined that any two fault data are correlated in time.

[0027] The device correlation is determined by cross-comparing the sets of devices involved in the fault data: Physical connections or logical dependencies between devices are obtained from the big data network module database, including but not limited to devices in the same cabinet, under the same router, within the same subnet, or upstream and downstream devices with data dependencies. The sets of devices involved in two fault data sets are compared one by one, and the number of device pairs with topological relationships is counted to obtain the effective correlation count. The larger value of the two device set sizes is used as a normalization factor, and the effective correlation count is then compared with this normalization factor to obtain a device correlation index between zero and one. The closer the device correlation index value is to one, the closer the connection between the devices involved in the two sets of fault data. The calculated device correlation index value is compared with a set device correlation threshold. If the device correlation index is greater than or equal to the device correlation threshold, it is determined that the two sets of fault data are correlated on the device. The set device correlation threshold defaults to 0.4, and its value is dynamically adjusted according to the number of devices in the system.

[0028] If two existing fault data points are correlated in time and related on the device, a corresponding fault scenario record is generated. Each fault scenario record includes information such as a set of fault types, fault occurrence conditions, impact scope, and estimated risk level. Finally, the constructed fault scenario records are statistically analyzed to obtain a fault scenario set, which is then stored in the database of the big data network module.

[0029] The specific process of using a computer data terminal to collect operational data and perform anomaly analysis is as follows: The computer data terminal collects operational data in real time according to a set sampling frequency, including but not limited to CPU utilization, memory usage, disk I / O rate, network throughput, process running status, and key log information. After collection, the raw operational data is preprocessed to obtain operational data. After preprocessing, the operational data is mapped to a multi-dimensional feature space to obtain multi-dimensional features. Then, the multi-dimensional features are compared with the baseline features of historical normal operation datasets stored in the big data network module. Features exceeding the threshold of a single or combined baseline feature are initially marked as anomalies. The baseline features include, but are not limited to, numerical range, fluctuation amplitude, change trend, correlation coefficient, and time series periodicity.

[0030] Simultaneously, a multi-model parallel detection mechanism is designed, including: using statistical methods to detect anomalies in the running data to obtain running numerical distribution; then using cluster analysis to identify the offset of the running state cluster centers, denoted as the running state offset; and combining the obtained running numerical distribution anomalies and running state offset with a machine learning model to identify potential anomalies under complex patterns, specifically: The abnormal distribution of running numerical values ​​and the deviation of running state are standardized. The standardized results are then fused with the corresponding feature vectors of the abnormal distribution of running numerical values ​​and the deviation of running state in chronological order to obtain a multi-dimensional feature input matrix. This multi-dimensional feature matrix is ​​input into a pre-trained machine learning model, such as an isolated forest model or a support vector machine anomaly detection model. During runtime, the machine learning model utilizes the feature boundaries of normal and abnormal patterns learned during training to perform pattern matching and deviation calculation on the input data. In the training phase, the machine learning model establishes feature boundaries based on historical normal and abnormal running data. Specifically, the isolated forest model constructs multiple random trees and calculates the average path length of sample points in the tree structure; a shorter path indicates a more robust sample. Isolated forests are easily isolated, corresponding to a higher probability of anomalies. Support vector machine (SVM) anomaly detection models establish hyperplane boundaries by learning the feature distribution of normal data, using the distance from input sample points to the hyperplane as a measure of deviation. During runtime, the multidimensional feature samples to be detected are input into the model, and the model calculates their matching degree under the learned feature boundaries: for isolated forests, the average path length of the samples is obtained and normalized to obtain the deviation; for SVMs, the distance from the sample to the boundary hyperplane is obtained and scaled to obtain the feature deviation. When the feature deviation exceeds the feature deviation threshold set by the machine learning model, the feature sample is determined to be a potential anomaly, and anomaly feature labels and corresponding confidence scores are output as anomalous data features in complex patterns, and a set of anomalous feature vectors is obtained.

[0031] The fault analysis module obtains the set of abnormal feature vectors and the set of fault scenarios for diagnosis to obtain the initial fault diagnosis, specifically: The set of abnormal feature vectors will be obtained, denoted as E = {e1, e2, ..., e...}. i1 , ..., e n1 Let the set of obstacle scene records be S = {s1, s2, ..., s}. j1 , ..., s m1 Further diagnostic procedures employ a two-stage algorithm model, including: The FAISS vector retrieval algorithm is used to construct a fault scene vector index for each abnormal feature vector e. i1 Retrieve the top-K nearest neighbor fault scenario vectors s from the index. j1 The similarity metric uses cosine similarity to obtain the similarity score Sim(e). i1 ,s j1 ), and the similarity Sim(e i1 ,s j1 The similarity threshold θs of the candidate scene is compared with the similarity threshold θs. If the similarity is Sim(e) i1 ,s j1If the similarity is greater than the similarity threshold θs, then it is considered a candidate scene. Based on the obtained candidate scenarios, the features of the candidate scenarios are concatenated with the anomaly feature vectors to form the input feature matrix H, which is then input into the Gradient Boosting Tree (GBDT) model. During the training phase, the model has learned the mapping relationship between historical anomalies and fault scenarios. Based on this mapping relationship, the model outputs the probability distribution of fault types corresponding to each candidate scenario, P(fault type...). k |e i1 Then, the fault type with the highest probability is selected as the initial fault scenario set.

[0032] The fault handling module, based on the obtained initial fault scenario set, performs evolution simulation on each initial fault scenario to obtain multi-directional offset fault scenarios; specifically: Starting from the initial set of fault scenarios, a multi-directional evolutionary simulation algorithm is used to simulate each initial scenario under various external perturbations, including load fluctuation index ΔL, network latency index ΔN, equipment state change index ΔD, and task scheduling adjustment index ΔTs, to generate a set of offset fault scenarios. For each offset scenario, a multi-dimensional feature matrix is ​​constructed and normalized. The normalized feature matrix is ​​then input into a Long Short-Term Memory (LSTM) network, which updates the hidden state and unit state at each time step t. Finally, the event type probability at each time step is obtained through a Softmax output layer. The event type probability is compared with an event type probability threshold; if the event type probability is greater than the threshold, it is recorded as a high-confidence event.

[0033] For trend indicators, including temperature and latency, the Prophet model decomposes them into trend g(t1), seasonality s(t1), and residual h(t1): i.e., trend indicators Then, the event sequence predicted by the Long Short-Term Memory Network is time-aligned with the Prophet prediction metric to generate the offset fault scenario path, i.e., Path(c i2 ′)={e1, e2, ..., e k1}, e k1 =(event_type,t k1 D k1 ), where D k1 This refers to the set of affected devices.

[0034] The fault early warning module obtains the offset fault scene path based on the multi-directional offset fault scenes, and calculates the offset of each offset fault scene, specifically: The Dynamic Time Warping (DTW) algorithm is used to align the offset path with the original scene path, using the formula: The offset is calculated. Where K1 is the number of events in the aligned path, and dist(...) is a measure of the difference between two corresponding events, defined as: , where α·1 type If the event type is different, it is recorded as 1; otherwise, it is recorded as 0. Multiplying by the weighting coefficient α indicates the importance of the type difference. This represents the proportion of the calculation time difference to the preset window T1; This indicates the similarity of the sets of affected devices. If the devices affected by two events are almost identical, the value is close to 1; otherwise, it is 0. The complexity of the fault is determined based on the offset; then, the preprocessing time for the fault is determined based on the complexity of the fault. The process is as follows: Based on the calculated offset Constructing the complexity vector z i2 =[Offset(ci2′), RL(ci2′), DI(ci2′), PS(ci2′), Uy(ci2′)]; where Offset(ci2′), RL(ci2′), DI(ci2′), and PS(ci2′) correspond to path deviation, risk level normalization value, proportion of affected devices, and event arrival rate, respectively; where: the risk level normalization value is obtained from the fault impact assessment library of the big data network module; the proportion of affected devices is obtained by calculating the number of affected devices divided by the total number of devices in the network or system based on the device status monitoring data of the real-time monitoring platform; the event arrival rate is extracted from the time series of monitoring indicators and the number of new fault alarms occurring per unit time is counted; and then the fault complexity is obtained through weighted linear-nonlinear fusion. : ,in, The weights w are obtained by Bayesian optimization from historical fault samples, representing the linear weighted contributions of each individual indicator. This refers to the interaction terms between different indicators, capturing the combined effect of risk factors; the coefficient η ab Also originating from the training process; σ is the activation function, using Logistic: mapping the results to a complexity probability range of 0 to 1; then calculating the complexity value... Then input into the duration prediction model f θ (·) Obtain the preprocessing time: T pre =f θ ( u i2 ), where u i2 The context feature vector is obtained from the database within the big data network module.

[0035] Fault-based preprocessing time T preThen, obtain the set of tasks currently being executed by the computer; R = {r1, r2, ..., r...} i3 , ..., r n3}, and extract its corresponding execution path to form a task path set P. R ; Set the processing path Path(c) for the current offset fault scenario i2 ′) and task path set P R Perform conflict detection, specifically as follows: For r i3 Task path P Ri3 With offset fault path Path(c i2 If time intervals overlap and the execution node sets intersect, then a conflict is considered to have occurred, i.e., the conflict state value is: , where Δt overlap Indicates the length of overlap between time intervals. and Let r and r represent the sets of computation nodes involved in the path, respectively; if for any task r i3 Conflict state value If a conflict occurs, the offset fault scenario is marked as a conflict offset fault scenario, and a fault warning is triggered. If no conflict exists, the process proceeds to the next round of offset fault scenario evolution, including: based on the current scenario, obtaining a new batch of offset scenarios through the above-mentioned process of simulating the development of each initial fault scenario to obtain multi-directional offset fault scenarios; similarly, the conflict state value is recalculated for each new scenario. The iteration will terminate if any of the following conditions occur: all generated scenes conflict and an early warning has been triggered, or the scene evolution reaches the set limit. The process of all generated scenarios being in conflict and triggering an early warning is as follows: For all offset fault scenarios generated in the current iteration batch, calculate the conflict value based on the task conflict detection formula. If the conflict value of each offset fault scenario is equal to one, then all generated scenarios are considered to be in conflict and no further evolution will be performed. The process of the scenario evolution reaching the set upper limit is as follows: when the number of iterations that the current offset fault scenario has undergone reaches the preset evolution upper limit, no further evolution will be carried out.

[0036] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to any specific implementation. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A computer fault early warning system based on big data, comprising: The system comprises a big data network module, a computer data terminal, a fault analysis module, a fault handling module, and a fault early warning module; its features are: The big data network module contains various types of computer data, and a set of fault scenarios is obtained by constructing fault scenarios based on these various types of computer data. Computer data terminals are used to perform anomaly analysis on the collected operational data to obtain a set of anomaly feature vectors; The fault analysis module acquires a set of abnormal feature vectors and a set of fault scenarios for diagnosis; the set of fault scenarios is diagnosed using a two-stage algorithm model to obtain candidate scenarios; based on the obtained candidate scenarios, the initial fault scenario set is output within the gradient boosting tree (GBDT) model. The fault handling module simulates each initial fault scenario in the obtained initial fault scenario set to obtain multi-directional offset fault scenarios. The fault early warning module obtains the offset fault scenario path based on the multi-directional offset fault scenario and calculates the offset of each offset fault scenario; the complexity of the fault is calculated from the offset, and the preprocessing time of the fault is obtained based on the fault complexity; based on the fault preprocessing time, the set of task paths currently being executed by the computer is obtained, and the processing path of the current offset fault scenario is conflicted with the set of task paths to obtain the conflict status value.

2. The computer fault early warning system based on big data according to claim 1, characterized in that, The specific process of constructing fault scenarios based on various types of computer data is as follows: Computer fault type data is obtained from the Internet through web crawlers or API calls, and corresponding fault data is generated using association rule analysis algorithms. The derived fault data sets are combined and associated to form various fault scenarios.

3. The computer fault early warning system based on big data according to claim 2, characterized in that, The process of generating corresponding fault data using association rule analysis algorithms is as follows: The acquired fault type data is cleaned and standardized to construct a transaction set. Based on the transaction set, a minimum support threshold is set, and the Apriori algorithm is used to calculate the frequency of occurrence of each fault type and its combination in the transaction set. The minimum support threshold is dynamically adjusted according to the distribution of historical data, and fault type combinations with support greater than or equal to the minimum support threshold are selected to obtain frequent itemsets. The frequent itemsets are then sorted according to the size of their support. For each frequent itemset, candidate rules are generated, and the confidence level of each candidate rule is calculated. Based on the confidence level of the candidate rules, rules with a confidence level greater than or equal to the minimum confidence level threshold are selected as valid association rules. Based on the obtained valid association rules, the fault type events corresponding to the rules are mapped to the corresponding fault data attributes, and the fault data set is obtained statistically.

4. A computer fault early warning system based on big data according to claim 2, characterized in that, The specific process of combining and associating the derived fault data set is as follows: Matching is performed based on the time correlation and device correlation between fault data: the time correlation is calculated by comparing the occurrence time of each fault data in the fault data set and calculating the time difference between any two fault data. When the time difference is less than a preset time window, it is determined that any two fault data are correlated in time. The device correlation is determined by cross-comparing the device sets involved in the fault data: the device sets involved in the two fault data are compared one by one, the number of device pairs with topological relationships is counted, and the device correlation index is calculated based on the size of the combined device sets. The calculated device correlation index is compared with a set device correlation threshold. If the device correlation index is greater than or equal to the device correlation threshold, it is determined that the two fault data are correlated on the device, and a corresponding fault scene record is generated.

5. A computer fault early warning system based on big data according to claim 1, characterized in that, The process by which a computer data terminal collects operational data and performs anomaly analysis is as follows: The computer data terminal collects operational data in real time according to the set sampling frequency; the raw operational data is preprocessed to obtain operational data; the operational data is mapped to a multidimensional feature space to obtain multidimensional features; the multidimensional features are then compared with the historical normal operation dataset stored in the big data network module to obtain baseline features; and features that exceed the threshold of a single or combined baseline feature are initially marked as anomalies. The designed multi-model parallel detection mechanism includes: using statistical methods to detect anomalies in the running data to obtain running numerical distribution anomalies; then using cluster analysis to identify the offset of the running state cluster centers, denoted as the running state offset; and combining the obtained running numerical distribution anomalies and the running state offset with a machine learning model to identify potential anomalies in complex patterns, specifically: The abnormal distribution of running numerical values ​​and the offset of running state are standardized; the standardized results are fused with the feature vectors of the corresponding abnormal distribution of running numerical values ​​and the offset of running state in chronological order to obtain a multi-dimensional feature input matrix; the multi-dimensional feature matrix is ​​input into a pre-trained machine learning model to obtain a set of abnormal feature vectors.

6. A computer fault early warning system based on big data according to claim 1, characterized in that, The process of diagnosing fault scenario records using a two-stage algorithm model: The FAISS vector retrieval algorithm is used to construct a fault scene vector index. For each abnormal feature vector, the top-K nearest neighbor fault scene vectors are retrieved in the index. The similarity is measured by cosine similarity. The similarity is compared with the similarity threshold corresponding to the candidate scene. If the similarity is greater than the similarity threshold, it is the corresponding candidate scene. Based on the obtained candidate scenarios, the features of the candidate scenarios are concatenated with the abnormal feature vectors to form the input feature matrix, which is then input into the Gradient Boosting Tree (GBDT) model. The model has learned the mapping relationship between historical anomalies and fault scenarios during the training phase. Based on the mapping relationship, the fault type probability corresponding to each candidate scenario is output, and the fault type corresponding to the highest probability is taken as the initial fault scenario set.

7. A computer fault early warning system based on big data according to claim 1, characterized in that, The process of obtaining the offset fault scene path based on the multi-directional offset fault scene is as follows: Starting from the initial set of fault scenarios, a multi-directional evolution simulation algorithm is used to generate a set of offset fault scenarios for each initial scenario under various external perturbations. A multi-dimensional feature matrix is ​​constructed for each offset scenario and input into a long short-term memory network. The long short-term memory network updates the hidden state and unit state at each time step. Finally, the event type probability at each time step is obtained through the softmax output layer. The event type probability is compared with the event type probability threshold. If the event type probability is greater than the event type probability threshold, it is recorded as a high-confidence event. For trend indicators, the Prophet model is used to decompose the trend indicator into trend, seasonality and residual. Next, the event sequence predicted by the Long Short-Term Memory Network is time-aligned with the Prophet prediction index to generate an offset fault scene path; based on the offset fault scene path, a dynamic time warping algorithm is used to align the offset path with the original scene path to obtain the offset amount.

8. A computer fault early warning system based on big data according to claim 1, characterized in that, The specific process of calculating the fault complexity from the offset and then determining the fault preprocessing time based on the fault complexity is as follows: Based on the calculated offset, a complexity vector is constructed, and the fault complexity is obtained through weighted linear-nonlinear fusion. The complexity value is then input into the duration prediction model to obtain the preprocessing duration.

9. A computer fault early warning system based on big data according to claim 1, characterized in that, The specific process of performing conflict detection between the processing path of the current offset fault scenario and the task path set is as follows: Obtain the set of tasks currently being executed by the computer, extract the corresponding execution paths, and form a task path set; perform conflict detection between the processing path of the current offset fault scenario and the task path set, specifically: For task paths and offset fault paths, if the time intervals overlap and the execution node sets intersect, a conflict is considered. The conflict state value is obtained through a formula. If the conflict state value is one for any task, the offset fault scenario is marked as a conflict offset fault scenario, and a fault warning is triggered. If no conflict exists, the process proceeds to the next round of offset fault scenario evolution, including: Based on the current scenario, the process of simulating the development of each initial fault scenario to obtain multi-directional offset fault scenarios is used to obtain a new batch of offset scenarios. Similarly, the conflict state value is recalculated for each new scenario. The iteration terminates when any of the following situations occur: all generated scenarios are in conflict and a warning has been triggered, or the scenario evolution reaches the set upper limit.

Images