Real-time analysis and fault positioning system for big data

Abstract

The invention discloses a big-data-oriented real-time analysis and fault positioning system, and the system comprises a data collection module which is used for collecting a multi-source data stream, and constructing a high-dimensional feature data set; the confrontation feature compression module is used for compressing high-dimensional features through multi-layer nested mapping and generating initial potential feature representation; the sparrow population module is used for constructing a sparrow individual population containing a plurality of hyper-parameter combinations; the feature evolution analysis module is used for deploying the optimized analysis confrontation model to a big data platform; the abnormal event detection module is used for carrying out sliding window analysis based on the change amplitude and the deviation trend of the potential characteristic track in the time sequence and judging whether a real-time abnormal event is formed or not; the historical alignment matching module is used for marking a specific time index and a source channel of fault occurrence; and the label identification module is used for completing fault type identification and fault type label output based on the fault occurrence time index and the source channel. According to the invention, an accurate big data real-time analysis and fault positioning scheme is provided for the user.

Description

Technical Field

[0001] The present invention relates to the technical field of big data fault diagnosis, and in particular to a real-time analysis and fault location system for big data. Background Art

[0002] Against the backdrop of the rapid development of informatization and industrial intelligence, along with the widespread deployment of technologies such as the Industrial Internet of Things, cloud computing, and big data, the real-time collection and processing of massive amounts of multi-source data has become a critical foundation for ensuring enterprise operations and system stability. In typical application scenarios such as intelligent manufacturing, smart energy, financial risk control, and cloud platform operations and maintenance, the operation of business systems and underlying hardware systems continuously generates multidimensional data streams, including industrial sensor data, operational logs, performance metrics, and historical fault records. Promptly identifying potential operational anomalies, fault symptoms, and system-level resource bottlenecks from this complex, heterogeneous data has become a key research topic in big data processing and intelligent operations and maintenance.

[0003] Existing fault diagnosis and anomaly detection methods are primarily based on rule matching, statistical analysis, or rely on traditional machine learning models such as support vector machines, decision trees, and K-means clustering. These methods can achieve certain results in specific data scenarios, but they have significant limitations in complex real-world environments. First, traditional methods rely on manual feature extraction and struggle to adapt to dynamically changing, high-dimensional, and complex data. In particular, their generalization capabilities significantly decrease when processing large-scale data streams with time-series correlations and multi-channel heterogeneous inputs. Second, existing methods often rely on static models, making it difficult to update them in real time and unable to effectively capture gradual changes in system operating states and sudden anomalies. Furthermore, some methods fail to consider the expression patterns of data in the latent space, resulting in insufficient model representation capabilities and a weak ability to recognize complex, nonlinear patterns such as system-level resource exhaustion faults and distributed anomalies.

[0004] In recent years, with the development of deep learning technology, structures such as autoencoders, variational encoders, and generative adversarial networks have been widely introduced into the fields of anomaly detection and fault identification. Among them, the adversarial autoencoder model has become one of the core tools for high-dimensional anomaly detection due to its excellent ability to extract nonlinear features and model latent space distributions. However, in practical applications, the parameter configuration of adversarial autoencoder models relies heavily on empirical settings and lacks a systematic automatic optimization mechanism, resulting in large fluctuations in model performance across different scenarios. Furthermore, most related research has neglected intelligent search mechanisms during the model training process, making it difficult to achieve a globally optimal hyperparameter combination, thus affecting the final detection accuracy and robustness.

[0005] The sparrow search algorithm, an emerging swarm intelligence optimization algorithm, simulates the foraging, vigilance, and escape behaviors of sparrow flocks to perform parameter searches. It boasts a simple structure, fast convergence, and strong global search capabilities. However, the original algorithm's fixed role partitioning mechanism and behavioral strategy make it difficult to adapt to dynamic changes in the feature space and model performance feedback. This leads to a tendency to fall into local optimality during high-dimensional parameter optimization and lacks stability. Furthermore, existing algorithms focus solely on the output of abnormal events, such as result classification or alarms, lacking trajectory modeling and spatial interpretation of the anomaly generation process. This makes it difficult to meet the growing demand for anomaly tracing, root cause location, and strategy optimization in industrial applications.

[0006] Existing solutions for subsequent processing of anomaly detection results primarily rely on independent judgment, ignoring the structural correlation between anomaly trajectories and historical fault samples. They also lack an alignment mechanism based on multi-dimensional feature fusion, making it impossible to accurately locate and label system-level anomalies. Traditional approaches to model updates generally employ static training processes, making it difficult to develop continuous learning capabilities for real-time operating environments. This is particularly true when faced with system topology changes or business policy adjustments, resulting in rapid degradation of model performance and a lack of adaptive closed-loop mechanisms.

[0007] Therefore, how to provide a real-time analysis and fault location system for big data is an urgent problem that those skilled in the art need to solve. Summary of the Invention

[0008] One purpose of the present invention is to propose a real-time analysis and fault location system for big data. The present invention performs feature compression through an adversarial autoencoder model, combines it with an improved sparrow search algorithm to optimize the parameters of the analysis adversarial model, detects real-time abnormal events based on the evolutionary analysis of potential feature trajectories, further realizes fault location and type recognition through historical sample alignment and matching, and constructs a model online update mechanism to form an intelligent fault handling solution that integrates anomaly detection, fault identification and adaptive optimization of the analysis adversarial model.

[0009] A real-time analysis and fault location system for big data according to an embodiment of the present invention includes:

[0010] Data acquisition module, used to collect multi-source data streams and build high-dimensional feature data sets;

[0011] Adversarial feature compression module, which is used to compress high-dimensional features through multi-layer nested mapping to generate initial latent feature representation;

[0012] The sparrow population module is used to construct a population of sparrow individuals containing multiple hyperparameter combinations;

[0013] Feature evolution analysis module, used to deploy the optimized analysis and adversarial model to the big data platform;

[0014] The abnormal event detection module is used to perform sliding window analysis based on the change amplitude and offset trend of potential feature trajectories in the time series to determine whether a real-time abnormal event occurs;

[0015] The history alignment and matching module marks the specific time index and source channel of the fault;

[0016] The label identification module is used to complete fault type identification and fault type label output based on the time index and source channel of the fault.

[0017] Optionally, modules can be connected using the following methods:

[0018] S1, collect multi-source data streams and build high-dimensional feature datasets;

[0019] S2. Based on the high-dimensional feature dataset, an adversarial autoencoder model is used to build an analytical adversarial model to generate a low-dimensional latent feature representation.

[0020] S3. Optimize the parameters of the analytical adversarial model using an improved sparrow search algorithm, construct a population of sparrow individuals to represent different hyperparameter combinations, and perform a global search and update the analytical adversarial model by simulating foraging, vigilance, and escape behaviors;

[0021] S4. Deploy the optimized analysis adversarial model in a big data processing platform, receive input data streams in real time, and generate corresponding low-dimensional latent feature representations and latent feature trajectories;

[0022] S5. Detect real-time abnormal events based on trajectory changes and potential feature trajectory deviations represented by low-dimensional latent features. For system-level resource exhaustion faults, output abnormal trajectories.

[0023] S6. Combine the abnormal trajectory and the corresponding low-dimensional latent feature representation as the abnormal event feature, align and match them with the feature trajectory of historical fault samples, and mark the specific time index and source channel of the fault;

[0024] S7. Based on the specific time index and source channel of the fault, further identify and label the fault type, and collect abnormal recognition data during operation and update the analysis adversarial model.

[0025] Optionally, the multi-source data stream includes industrial sensor monitoring data, business processing logs, system operation indicators and historical fault records.

[0026] Optionally, the anomaly identification data includes prediction deviation, classification error and actual alarm records.

[0027] Optionally, S2 includes the following specific steps:

[0028] S21. Initialize the structural parameters of the adversarial autoencoder model based on the high-dimensional feature dataset;

[0029] S22. Combine the adversarial autoencoder model and the high-dimensional feature dataset to build an analytical adversarial model, and input the high-dimensional features as input samples into the adversarial autoencoder model;

[0030] S23, input the input samples into the encoder in sequence, compress the high-dimensional features through multi-layer nested mapping, and generate the initial potential feature representation;

[0031] S24, inputting the initial latent feature representation into the decoder, generating corresponding reconstructed samples, and calculating the reconstruction error between the original input samples and the reconstructed samples;

[0032] S25, input the reconstruction error and the initial potential feature representation output by the encoder into the discriminator unit, optimize the parameters of the encoder and decoder through the adversarial training mechanism, and update the analysis adversarial model;

[0033] S26. Use the analytical adversarial model to represent and compress high-dimensional feature data and output a stable low-dimensional latent feature representation.

[0034] Optionally, S3 includes the following specific steps:

[0035] S31. Combine the analysis of the adversarial model and the low-dimensional latent feature representation to initialize the population of sparrow individuals, where each sparrow individual represents a hyperparameter combination.

[0036] S32. Based on the hyperparameter combination corresponding to each individual sparrow, configure an analysis adversarial model and train it on a high-dimensional feature dataset to obtain performance evaluation indicators of the analysis adversarial model corresponding to each individual sparrow;

[0037] S33, ranking the fitness of sparrow individuals according to the performance evaluation index of the analysis adversarial model, and dividing them into foragers and guards;

[0038] S34. By simulating the three-stage action of sparrow foraging behavior, a position update strategy is implemented on foragers, a random perturbation operation is performed on guarders, and an escape migration operation is performed on some marginal individuals to generate a new generation of sparrow individual groups. Compared with the sparrow search algorithm, the improved sparrow search algorithm introduces an escape direction adjustment strategy based on global distribution feedback. The escape migration direction of marginal sparrow individuals has clustering guidance, which reduces the offset loss caused by random escape:

[0039]

[0040] in, represents the hyperparameter position vector of the i-th sparrow individual at the t-th iteration, represents the hyperparameter position vector of the i-th sparrow individual after update at the t+1th iteration, α represents the global search step adjustment factor, N represents the total number of sparrow individuals in the sparrow population, represents the hyperparameter vector position of the j-th sparrow individual at the t-th iteration, ||·|| represents the Euclidean distance, ∈ represents a small constant to prevent the denominator from being zero, and w j represents the fitness weight of the j-th sparrow individual, i represents the number of the sparrow individual whose position is currently being updated, j represents the reference individual number used to guide the update of the i-th sparrow individual, and t represents the number of the current search iteration round;

[0041] S35. Continue configuring the hyperparameter combination of the new generation of sparrow individuals into the analysis and adversarial model, and repeat the training and evaluation process to form a global search mechanism based on the sparrow foraging behavior until the total number of iterations reaches 100 rounds;

[0042] S36. Select sparrow individuals with a reconstruction error less than 0.015 from the final iterative sparrow individual population, and update the analysis adversarial model based on the hyperparameter combination corresponding to the sparrow individuals with a reconstruction error less than 0.015.

[0043] Optionally, S4 includes the following specific steps:

[0044] S41. Deploy the analysis adversarial model to the model running environment of the big data processing platform;

[0045] S42, receiving a continuously input data stream in the analysis adversarial model running environment, and converting the data stream into a standardized high-dimensional feature data set according to a structured preprocessing rule;

[0046] S43. Input the standardized high-dimensional feature dataset into the encoder of the deployed analytical adversarial model to generate a real-time low-dimensional latent feature representation sequence:

[0047]

[0048] in, represents the low-dimensional potential feature representation of the nth sample at time step τ, k represents the layer number of the encoder, n represents the data sample number, τ represents the time step number, Represents the high-dimensional feature vector of the nth sample at time step τ, Indicates the optimal model parameter combination The k-th layer encoder weight matrix is determined by Indicates the optimal model parameter combination The bias vector determined by K is the total number of encoder modules, i is the number of the individual sparrow whose position is currently being updated, and t is the number of the current search iteration round.

[0049] S44, inputting the low-dimensional latent feature representation into the decoder, generating real-time reconstructed samples, and calculating a new reconstruction error based on the input data and the reconstructed samples;

[0050] S45. Stack the continuously generated low-dimensional latent feature representations in time series, extract the changing trends between features over time, and construct latent feature trajectories;

[0051] S46. Jointly output the low-dimensional latent feature representation, the new reconstruction error, and the latent feature trajectory.

[0052] Optionally, S5 includes the following specific steps:

[0053] S51. Based on the low-dimensional latent feature representation and latent feature trajectory, the trajectory evolution sequence is constructed in chronological order;

[0054] S52, constructing multiple sets of overlapping sliding windows based on the trajectory evolution sequence, and synchronously extracting trajectory change direction and amplitude information within each set of windows;

[0055] S53, identifying the trajectory direction mutation behavior and the unstable fluctuation of feature expression in each overlapping window, and performing consistency check on the window intersection results to screen the candidate segments of the real trajectory change;

[0056] S54, performing weighted fusion on the offset path length, offset duration, and change density of the trajectory in the candidate segment of the real trajectory change to construct an anomaly priority scoring matrix;

[0057] S55. A dynamic judgment threshold is set based on the abnormal priority scoring matrix to determine whether a real-time abnormal event is constituted, and all trajectory segments that are below the threshold but appear continuously are stored in the abnormal memory vector pool:

[0058]

[0059] in, represents the abnormal score of the nth sample at time step τ, represents the k-th dimension component in the low-dimensional latent feature representation of the n-th sample at time step τ, D represents the total number of dimensions of the low-dimensional latent feature vector, W represents the length of the lookback window, w represents the historical time offset step within the window, n represents the data sample number, τ represents the time step number, and d represents the dimension number of the d-th dimension component in the latent feature vector;

[0060] S56. Output the confirmed real-time abnormal event as a system-level resource exhaustion fault abnormal trajectory, and perform comparison and analysis on the trajectory pattern in the abnormal memory vector pool to output the final abnormal trajectory.

[0061] Optionally, S6 includes the following specific steps:

[0062] S61. Combining the final abnormal trajectory with the corresponding low-dimensional latent feature representation, constructing an abnormal event feature set;

[0063] S62, representing a template of feature trajectories in the historical fault sample set and corresponding low-dimensional latent features;

[0064] S63, performing structural alignment on the trajectory pattern sequence in the abnormal event feature set and the characteristic trajectory in the historical fault sample to generate a trajectory similarity matrix;

[0065] S64, performing vector comparison analysis on the low-dimensional potential feature expression sequence in the abnormal event feature set and the feature trajectory in the historical sample to generate a feature representation similarity matrix;

[0066] S65. Perform weighted fusion of the trajectory similarity matrix and the feature representation similarity matrix to construct a multi-dimensional matching score matrix. Calculate the weighted scores of each historical fault sample and the abnormal event feature in the trajectory similarity and feature vector similarity dimensions, with weight ratios of 0.6 and 0.4. Take the historical sample with the highest total score of the weighted scores of the trajectory similarity and feature vector similarity dimensions and a score of not less than 0.85 as the most matching historical fault sample.

[0067] S66. Extract the fault time index and source channel from the most matching historical fault sample and mark them as the positioning result of the current abnormal event feature.

[0068] Optionally, S7 includes the following specific steps:

[0069] S71. Extract the latest low-dimensional potential feature representation within the positioning window based on the fault time index and source channel, and matching with the timestamp and channel identifier in the current input data stream;

[0070] S72. The latest low-dimensional potential feature representation is combined with the corresponding time index and source channel information to form the latest joint feature vector, and similarity matching is performed with the abnormal event features marked in the historical fault sample library. If the similarity is higher than 0.88 and the time index of the current abnormal event is offset from the time of the historical fault sample by no more than 3 time steps, it is determined to be the fault type corresponding to the historical sample, and the corresponding fault type label is output;

[0071] S73: Bind the fault type label with the time index and the source channel to form a structured annotation result and output it to the label recording unit;

[0072] S74. Perform a cross-check on the final abnormal trajectory and the current fault type label, and extract samples with high consistency as update data;

[0073] S75. Input the updated data into the deployed analysis adversarial model to generate an updated analysis adversarial model parameter set;

[0074] S76. Perform a stability evaluation on the updated analysis adversarial model parameter set. If the preset performance standard is met, replace the parameter configuration of the current analysis adversarial model to complete the online adaptive update process of the analysis adversarial model.

[0075] The beneficial effects of the present invention are:

[0076] The present invention proposes a real-time analysis and fault location system for big data, which can effectively solve the shortcomings of existing technologies in high-dimensional data modeling, anomaly detection accuracy, fault location accuracy, and sustainable model updates, and significantly improve the system's intelligent analysis capabilities and stable operation guarantee capabilities. By introducing the adversarial autoencoder model, the present invention can automatically extract low-dimensional potential feature representations from complex data. Compared with the traditional method that relies on manually designed features, it has stronger adaptability and generalization in expressing nonlinear time series structures and abnormal patterns. At the same time, by constructing an improved sparrow search algorithm, simulating multi-stage behaviors such as foraging, vigilance, and escape, and realizing the global optimal search path in the parameter space, compared with static parameter settings or traditional optimization methods, its optimization accuracy is higher and the convergence speed is faster, which effectively guarantees the training effect and anomaly recognition performance of the analysis adversarial model.

[0077] The present invention can realize dynamic perception and processing of real-time data streams in the model deployment stage, and combined with the time evolution pattern analysis of potential feature trajectories, it has a strong ability to identify abnormal trends, and can detect potential risks such as system-level resource exhaustion before a fault occurs, providing an early warning mechanism for operation and maintenance management. On the basis of anomaly detection, a fault location method based on trajectory alignment and feature matching is further constructed, and abnormal events and historical fault samples are fused and compared in multiple dimensions. It can not only output accurate fault time indexes and source channels, but also improve the explainability and traceability of fault identification, meeting the dual requirements of accuracy and transparency in industrial big data applications. In addition, the present invention collects anomaly identification data during operation, combines the fault type label results, performs micro-batch updates and online replacements on the analysis adversarial model, and constructs a closed-loop model adaptation mechanism to achieve long-term maintenance and dynamic evolution of model performance, avoiding model aging and failure problems.

[0078] To sum up, the present invention realizes a full-process intelligent fault handling system from data collection, feature compression, intelligent optimization, anomaly detection, fault location to model update. It has the significant advantages of high modeling accuracy, strong optimization capability, high detection sensitivity, high positioning accuracy and strong continuous adaptability. It can be widely used in complex systems such as industrial Internet, smart energy, financial risk control, information system operation and maintenance that have high requirements for real-time analysis of big data and anomaly control, and has good technology promotion value and engineering application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0080] Figure 1 This is a method flow chart of a real-time analysis and fault location system for big data proposed by the present invention;

[0081] Figure 2 This is a system flow chart of a real-time analysis and fault location system for big data proposed by the present invention;

[0082] Figure 3 This is a data flow diagram of a real-time analysis and fault location system for big data proposed by the present invention. DETAILED DESCRIPTION

[0083] The present invention will now be described in further detail with reference to the accompanying drawings, which are simplified schematic diagrams that illustrate the basic structure of the present invention in a schematic manner.

[0084] refer to Figure 1-3 , a real-time analysis and fault location system for big data, including:

[0085] Data acquisition module, used to collect multi-source data streams and build high-dimensional feature data sets;

[0086] Adversarial feature compression module, which is used to compress high-dimensional features through multi-layer nested mapping to generate initial latent feature representation;

[0087] The sparrow population module is used to construct a population of sparrow individuals containing multiple hyperparameter combinations;

[0088] Feature evolution analysis module, used to deploy the optimized analysis and adversarial model to the big data platform;

[0089] The abnormal event detection module is used to perform sliding window analysis based on the change amplitude and offset trend of potential feature trajectories in the time series to determine whether a real-time abnormal event occurs;

[0090] The history alignment and matching module marks the specific time index and source channel of the fault;

[0091] The label identification module is used to complete fault type identification and fault type label output based on the time index and source channel of the fault.

[0092] This invention uses countermeasure feature compression and an improved sparrow search algorithm to extract latent feature representations based on multi-source data collection. It then integrates feature evolution analysis with a sliding trajectory detection mechanism to identify real-time abnormal events. Combined with historical alignment matching and label recognition modules, it locates fault time indexes and channels, enabling precise fault identification and label output, significantly improving fault location accuracy and response efficiency.

[0093] In this embodiment, the modules are connected through the following methods:

[0094] S1, collect multi-source data streams and build high-dimensional feature datasets;

[0095] S2. Based on the high-dimensional feature dataset, an adversarial autoencoder model is used to build an analytical adversarial model to generate a low-dimensional latent feature representation.

[0096] S3. Optimize the parameters of the analytical adversarial model using an improved sparrow search algorithm, construct a population of sparrow individuals to represent different hyperparameter combinations, and perform a global search and update the analytical adversarial model by simulating foraging, vigilance, and escape behaviors;

[0097] S4. Deploy the optimized analysis adversarial model in a big data processing platform, receive input data streams in real time, and generate corresponding low-dimensional latent feature representations and latent feature trajectories;

[0098] S5. Detect real-time abnormal events based on trajectory changes and potential feature trajectory deviations represented by low-dimensional latent features. For system-level resource exhaustion faults, output abnormal trajectories.

[0099] S6. Combine the abnormal trajectory and the corresponding low-dimensional latent feature representation as the abnormal event feature, align and match them with the feature trajectory of historical fault samples, and mark the specific time index and source channel of the fault;

[0100] S7. Based on the specific time index and source channel of the fault, further identify and label the fault type, and collect abnormal recognition data during operation and update the analysis adversarial model.

[0101] This paper optimizes the adversarial autoencoder model by improving the sparrow search algorithm, achieving high-dimensional feature compression and latent feature extraction from multi-source data. Combined with trajectory change detection and offset identification mechanisms, it can identify system-level anomalies in real time. By multi-dimensionally matching abnormal trajectories with historical samples, the fault time and channel are marked and the resulting labels are output. Simultaneously, a closed-loop self-updating model is established, improving fault identification accuracy and model adaptability.

[0102] In this embodiment, the multi-source data stream includes industrial sensor monitoring data, business processing logs, system operation indicators and historical fault records.

[0103] This method integrates industrial sensor monitoring data, business processing logs, system operation indicators, and historical fault records to construct a multi-source heterogeneous data feature space, improving the contextual integrity and feature expression diversity of fault analysis. A unified preprocessing mechanism achieves time series alignment and structural normalization, enhancing the model's robustness and generalization capabilities for dynamically changing scenarios, providing high-quality data support for subsequent potential feature extraction and anomaly identification.

[0104] In this embodiment, the abnormality identification data includes prediction deviation, classification error and actual alarm records.

[0105] This invention incorporates anomaly identification data, including prediction bias, classification error, and actual alarm records, to establish a multi-dimensional feedback mechanism for model updates. By dynamically integrating error behavior and alarm results, it enables quantitative assessment and fine-tuning of anomaly detection capabilities, improving the relevance and stability of the model's online learning. This enables the continuous self-adaptation of the analytical adversarial model, effectively enhancing fault identification accuracy and system robustness.

[0106] In this embodiment, S2 includes the following specific steps:

[0107] S21. Initialize the structural parameters of the adversarial autoencoder model based on the high-dimensional feature dataset;

[0108] S22. Combine the adversarial autoencoder model and the high-dimensional feature dataset to build an analytical adversarial model, and input the high-dimensional features as input samples into the adversarial autoencoder model;

[0109] S23, input the input samples into the encoder in sequence, compress the high-dimensional features through multi-layer nested mapping, and generate the initial potential feature representation;

[0110] S24, inputting the initial latent feature representation into the decoder, generating corresponding reconstructed samples, and calculating the reconstruction error between the original input samples and the reconstructed samples;

[0111] S25, input the reconstruction error and the initial potential feature representation output by the encoder into the discriminator unit, optimize the parameters of the encoder and decoder through the adversarial training mechanism, and update the analysis adversarial model;

[0112] S26. Use the analytical adversarial model to represent and compress high-dimensional feature data and output a stable low-dimensional latent feature representation.

[0113] This paper constructs an adversarial autoencoder structure comprising an encoder, decoder, and discriminator, performing multi-layer nested compression and reconstruction error-driven optimization on high-dimensional feature data to enhance the stability and discriminative power of the underlying feature representation. Combined with an adversarial training mechanism, the model parameters are continuously optimized, resulting in a compression result that retains key information while providing enhanced anomaly representation capabilities, providing a high-quality, low-dimensional feature foundation for fault detection and matching.

[0114] In this embodiment, S3 includes the following specific steps:

[0115] S31. Combine the analysis of the adversarial model and the low-dimensional latent feature representation to initialize the population of sparrow individuals, where each sparrow individual represents a hyperparameter combination.

[0116] S32. Based on the hyperparameter combination corresponding to each individual sparrow, configure an analysis adversarial model and train it on a high-dimensional feature dataset to obtain performance evaluation indicators of the analysis adversarial model corresponding to each individual sparrow;

[0117] S33, ranking the fitness of sparrow individuals according to the performance evaluation index of the analysis adversarial model, and dividing them into foragers and guards;

[0118] S34. By simulating the three-stage action of sparrow foraging behavior, a position update strategy is implemented on foragers, a random perturbation operation is performed on guarders, and an escape migration operation is performed on some marginal individuals to generate a new generation of sparrow individual groups. Compared with the sparrow search algorithm, the improved sparrow search algorithm introduces an escape direction adjustment strategy based on global distribution feedback. The escape migration direction of marginal sparrow individuals has clustering guidance, which reduces the offset loss caused by random escape:

[0119]

[0120] in, represents the hyperparameter position vector of the i-th sparrow individual at the t-th iteration, represents the hyperparameter position vector of the i-th sparrow individual after update at the t+1th iteration, α represents the global search step adjustment factor, N represents the total number of sparrow individuals in the sparrow population, represents the hyperparameter vector position of the j-th sparrow individual at the t-th iteration, ||·|| represents the Euclidean distance, ∈ represents a small constant to prevent the denominator from being zero, and wj represents the fitness weight of the j-th sparrow individual, i represents the number of the sparrow individual whose position is currently being updated, j represents the reference individual number used to guide the update of the i-th sparrow individual, and t represents the number of the current search iteration round;

[0121] S35. Continue configuring the hyperparameter combination of the new generation of sparrow individuals into the analysis and adversarial model, and repeat the training and evaluation process to form a global search mechanism based on the sparrow foraging behavior until the total number of iterations reaches 100 rounds;

[0122] S36. Select sparrow individuals with a reconstruction error less than 0.015 from the final iterative sparrow individual population, and update the analysis adversarial model based on the hyperparameter combination corresponding to the sparrow individuals with a reconstruction error less than 0.015.

[0123] This paper employs an improved sparrow search algorithm to construct a parameter optimization mechanism, conducting multiple rounds of global searches on the analytical adversarial model by simulating foraging, vigilance, and escape behaviors. An escape direction adjustment strategy based on global distributed feedback is introduced to guide the migration of marginal individuals into clusters, improving population convergence efficiency and search stability. A reconstruction error below 0.015 is used as the fitness screening criterion to ensure that the final model parameter combination has optimal expressiveness and anomaly detection performance.

[0124] In this embodiment, S4 includes the following specific steps:

[0125] S41. Deploy the analysis adversarial model to the model running environment of the big data processing platform;

[0126] S42, receiving a continuously input data stream in the analysis adversarial model running environment, and converting the data stream into a standardized high-dimensional feature data set according to a structured preprocessing rule;

[0127] S43. Input the standardized high-dimensional feature dataset into the encoder of the deployed analytical adversarial model to generate a real-time low-dimensional latent feature representation sequence:

[0128]

[0129] in, represents the low-dimensional potential feature representation of the nth sample at time step τ, k represents the layer number of the encoder, n represents the data sample number, τ represents the time step number, Represents the high-dimensional feature vector of the nth sample at time step τ, Indicates the optimal model parameter combination The k-th layer encoder weight matrix is determined by Indicates the optimal model parameter combination The bias vector determined by K is the total number of encoder modules, i is the number of the individual sparrow whose position is currently being updated, and t is the number of the current search iteration round.

[0130] S44, inputting the low-dimensional latent feature representation into the decoder, generating real-time reconstructed samples, and calculating a new reconstruction error based on the input data and the reconstructed samples;

[0131] S45. Stack the continuously generated low-dimensional latent feature representations in time series, extract the changing trends between features over time, and construct latent feature trajectories;

[0132] S46. Jointly output the low-dimensional latent feature representation, the new reconstruction error, and the latent feature trajectory.

[0133] This method deploys the optimized analytical adversarial model on a big data processing platform, receiving structured data streams in real time and generating low-dimensional latent feature representations. Dynamic representation quality assessment is achieved through decoding, reconstruction, and error analysis. By combining time series stacking to extract feature evolution trajectories and constructing latent spatial behavior patterns, this method provides high-resolution input support for anomaly detection and fault tracing, effectively enhancing the model's online representation capabilities and environmental adaptability.

[0134] In this embodiment, S5 includes the following specific steps:

[0135] S51. Based on the low-dimensional latent feature representation and latent feature trajectory, the trajectory evolution sequence is constructed in chronological order;

[0136] S52, constructing multiple sets of overlapping sliding windows based on the trajectory evolution sequence, and synchronously extracting trajectory change direction and amplitude information within each set of windows;

[0137] S53, identifying the trajectory direction mutation behavior and the unstable fluctuation of feature expression in each overlapping window, and performing consistency check on the window intersection results to screen the candidate segments of the real trajectory change;

[0138] S54, performing weighted fusion on the offset path length, offset duration, and change density of the trajectory in the candidate segment of the real trajectory change to construct an anomaly priority scoring matrix;

[0139] S55. A dynamic judgment threshold is set based on the abnormal priority scoring matrix to determine whether a real-time abnormal event is constituted, and all trajectory segments that are below the threshold but appear continuously are stored in the abnormal memory vector pool:

[0140]

[0141] in, represents the abnormal score of the nth sample at time step τ, represents the k-th dimension component in the low-dimensional latent feature representation of the n-th sample at time step τ, D represents the total number of dimensions of the low-dimensional latent feature vector, W represents the length of the lookback window, w represents the historical time offset step within the window, n represents the data sample number, τ represents the time step number, and d represents the dimension number of the d-th dimension component in the latent feature vector;

[0142] S56. Output the confirmed real-time abnormal event as a system-level resource exhaustion fault abnormal trajectory, and perform comparison and analysis on the trajectory pattern in the abnormal memory vector pool to output the final abnormal trajectory.

[0143] The present invention constructs an evolution sequence based on low-dimensional potential feature trajectories, uses multiple sets of overlapping sliding windows to extract trajectory direction mutations and fluctuation characteristics, integrates trajectory offset length and change density to generate an anomaly priority scoring matrix, dynamically sets judgment thresholds to identify real-time abnormal events, and introduces an anomaly memory vector pool to save subcritical trajectories, achieving early perception and accurate marking of system-level resource exhaustion failures, thereby improving the sensitivity and stability of anomaly detection.

[0144] In this embodiment, S6 includes the following specific steps:

[0145] S61. Combining the final abnormal trajectory with the corresponding low-dimensional latent feature representation, constructing an abnormal event feature set;

[0146] S62, representing a template of feature trajectories in the historical fault sample set and corresponding low-dimensional latent features;

[0147] S63, performing structural alignment on the trajectory pattern sequence in the abnormal event feature set and the characteristic trajectory in the historical fault sample to generate a trajectory similarity matrix;

[0148] S64, performing vector comparison analysis on the low-dimensional potential feature expression sequence in the abnormal event feature set and the feature trajectory in the historical sample to generate a feature representation similarity matrix;

[0149] S65. Perform weighted fusion of the trajectory similarity matrix and the feature representation similarity matrix to construct a multi-dimensional matching score matrix. Calculate the weighted scores of each historical fault sample and the abnormal event feature in the trajectory similarity and feature vector similarity dimensions, with weight ratios of 0.6 and 0.4. Take the historical sample with the highest total score of the weighted scores of the trajectory similarity and feature vector similarity dimensions and a score of not less than 0.85 as the most matching historical fault sample.

[0150] S66. Extract the fault time index and source channel from the most matching historical fault sample and mark them as the positioning result of the current abnormal event feature.

[0151] The present invention constructs an abnormal event feature set, fuses abnormal trajectories with potential feature representations, performs structural alignment and vector comparison with historical samples, generates trajectory similarity and feature representation similarity matrices respectively, and constructs a multidimensional matching score matrix through weighted fusion. It accurately extracts the best matching fault samples with a score of no less than 0.85, realizes abnormal event location based on the dual similarity of trajectory and feature, and improves the accuracy of fault identification and the credibility of judgment.

[0152] In this embodiment, S7 includes the following specific steps:

[0153] S71. Extract the latest low-dimensional potential feature representation within the positioning window based on the fault time index and source channel, and matching with the timestamp and channel identifier in the current input data stream;

[0154] S72. The latest low-dimensional potential feature representation is combined with the corresponding time index and source channel information to form the latest joint feature vector, and similarity matching is performed with the abnormal event features marked in the historical fault sample library. If the similarity is higher than 0.88 and the time index of the current abnormal event is offset from the time of the historical fault sample by no more than 3 time steps, it is determined to be the fault type corresponding to the historical sample, and the corresponding fault type label is output;

[0155] S73: Bind the fault type label with the time index and the source channel to form a structured annotation result and output it to the label recording unit;

[0156] S74. Perform a cross-check on the final abnormal trajectory and the current fault type label, and extract samples with high consistency as update data;

[0157] S75. Input the updated data into the deployed analysis adversarial model to generate an updated analysis adversarial model parameter set;

[0158] S76. Perform a stability evaluation on the updated analysis adversarial model parameter set. If the preset performance standard is met, replace the parameter configuration of the current analysis adversarial model to complete the online adaptive update process of the analysis adversarial model.

[0159] This method extracts low-dimensional latent features from the time index and channel corresponding to abnormal events, constructs a joint feature vector, and performs similarity matching with historical samples to identify the fault type. It also sets similarity thresholds and time offset constraints to achieve accurate label output. This method combines trajectory and label results for cross-verification, extracts consistent samples, and performs model updates, building a dynamic adaptive mechanism to improve fault identification accuracy and long-term model stability.

[0160] Example 1:

[0161] In order to verify the feasibility of the present invention in implementation, the present invention is applied to the data center infrastructure operation and maintenance platform of a large energy enterprise. The enterprise has multiple business nodes and computing sites across the country, and the large amount of sensor collection data, operation logs, resource scheduling indicators and service quality feedback data generated every day are uniformly imported into the data center. In the daily operation process, the system is often affected by multiple factors such as data center resource fluctuations, hardware aging, task conflicts, and sudden traffic. There are potential risks such as system load surges, memory leaks, and service response anomalies. However, the existing system mainly relies on fixed threshold warnings and manual rules. It is difficult to accurately identify early system anomalies, and it is even more impossible to locate system-level resource exhaustion failures in advance. Operation and maintenance personnel need to manually check after receiving the alarm. The average processing time is about 3.8 hours / time, which seriously affects the stability of the platform.

[0162] The real-time analysis and fault location system for big data proposed in the present invention is deployed, covering the entire process of data collection, feature modeling, anomaly identification, fault location and model update. The system first accesses 1,153 industrial sensors, 192 system log sources, 38 types of business processing service indicators and a total of 4,278 historical fault records in the past three years in the enterprise's ETL platform. Using the adversarial autoencoder model of the present invention, the input high-dimensional data is automatically compressed and feature extracted to construct a low-dimensional potential feature representation with a dimension of 64. The reconstruction error is maintained between 0.0192 and 0.0256 in the initial model. During the construction of the analysis model, the parameters of the encoder and decoder are jointly optimized by the improved sparrow search algorithm. The total number of iterations is 100 rounds. The reconstruction error of the individual with the best fitness reaches 0.0117, the KL divergence is 0.043, and the anomaly recognition accuracy of the model on the test set is improved to 97.2%.

[0163] After deploying the system, the platform began receiving real-time operational status data from the server cluster, processing an average of approximately 32,000 input samples per hour. By modeling the time series trajectory of low-dimensional latent features, the system can identify nonlinear patterns such as sustained spikes in CPU utilization and decreased memory release rates. At 4:40 a.m. on September 12, 2024, the system detected that the data service unit at node A17 had experienced a persistent deviation in the latent feature trajectory over the past eight time steps. Determined as severe by the anomaly priority scoring matrix, and combined with a historical sample alignment mechanism, the event was matched to a node failure sample caused by a cache overflow in July 2023, with a time index offset of +2 and 100% channel consistency. The system then output a "node-level resource exhaustion" fault type label at 4:43 a.m. Following the platform's prompts, operations and maintenance personnel quickly restarted some middleware modules and released the cache, avoiding service interruption. The entire response and processing process was completed within 7 minutes.

[0164] Table 1 Comparison of real-time analysis and fault location optimization effects for big data

[0165]

[0166]

[0167] Table 1 shows that after deploying the system of the present invention, the average monthly fault identification delay is 2.6 hours, the false alarm rate is 8.3%, the missed alarm rate is 12.7%, and the average location time of a typical system-level fault is 3.5 hours. After deploying the present invention, the average recognition delay of system anomalies dropped to 18 minutes, the false alarm rate dropped to 3.1%, the missed alarm rate dropped to 2.4%, the average fault location time was shortened to 23 minutes, and the location accuracy reached more than 95%, significantly improving the system's fault perception granularity and processing efficiency. At the same time, the model will automatically update once every 72 hours, and micro-batch incremental learning will be realized in combination with abnormal recognition data. In 7 consecutive cycles, the model reconstruction error has been stably maintained between 0.011 and 0.013, and there is no significant decrease in fitness.

[0168] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A real-time analysis and fault location system for big data, characterized by: include: Data acquisition module, used to collect multi-source data streams and build high-dimensional feature data sets; Adversarial feature compression module, which is used to compress high-dimensional features through multi-layer nested mapping to generate initial latent feature representation; The sparrow population module is used to construct a population of sparrow individuals containing multiple hyperparameter combinations; Feature evolution analysis module, used to deploy the optimized analysis and adversarial model to the big data platform; The abnormal event detection module is used to perform sliding window analysis based on the change amplitude and offset trend of potential feature trajectories in the time series to determine whether a real-time abnormal event occurs; The history alignment and matching module marks the specific time index and source channel of the fault; The label identification module is used to complete fault type identification and fault type label output based on the time index and source channel of the fault.

2. The real-time analysis and fault location system for big data according to claim 1, characterized in that: The modules are implemented as follows: S1, collect multi-source data streams and build high-dimensional feature datasets; S2. Based on the high-dimensional feature dataset, an adversarial autoencoder model is used to build an analytical adversarial model to generate a low-dimensional latent feature representation. S3. Optimize the parameters of the analytical adversarial model using an improved sparrow search algorithm, construct a population of sparrow individuals to represent different hyperparameter combinations, and perform a global search and update the analytical adversarial model by simulating foraging, vigilance, and escape behaviors; S4. Deploy the optimized analysis adversarial model in a big data processing platform, receive input data streams in real time, and generate corresponding low-dimensional latent feature representations and latent feature trajectories; S5. Detect real-time abnormal events based on trajectory changes and potential feature trajectory deviations represented by low-dimensional latent features. For system-level resource exhaustion faults, output abnormal trajectories. S6. Combine the abnormal trajectory and the corresponding low-dimensional latent feature representation as the abnormal event feature, align and match them with the feature trajectory of historical fault samples, and mark the specific time index and source channel of the fault; S7. Based on the specific time index and source channel of the fault, further identify and label the fault type, and collect abnormal recognition data during operation and update the analysis adversarial model.

3. The real-time analysis and fault location system for big data according to claim 2, characterized in that: The multi-source data stream includes industrial sensor monitoring data, business processing logs, system operation indicators and historical fault records.

4. The real-time analysis and fault location system for big data according to claim 2, characterized in that: The anomaly identification data includes prediction deviation, classification error and actual alarm records.

5. The real-time analysis and fault location system for big data according to claim 2, characterized in that: The S2 includes the following specific steps: S21. Initialize the structural parameters of the adversarial autoencoder model based on the high-dimensional feature dataset; S22. Combine the adversarial autoencoder model and the high-dimensional feature dataset to build an analytical adversarial model, and input the high-dimensional features as input samples into the adversarial autoencoder model; S23, input the input samples into the encoder in sequence, compress the high-dimensional features through multi-layer nested mapping, and generate the initial potential feature representation; S24, inputting the initial latent feature representation into the decoder, generating corresponding reconstructed samples, and calculating the reconstruction error between the original input samples and the reconstructed samples; S25, input the reconstruction error and the initial potential feature representation output by the encoder into the discriminator unit, optimize the parameters of the encoder and decoder through the adversarial training mechanism, and update the analysis adversarial model; S26. Use the analytical adversarial model to represent and compress high-dimensional feature data and output a stable low-dimensional latent feature representation.

6. The real-time analysis and fault location system for big data according to claim 2, characterized in that: The S3 includes the following specific steps: S31. Combine the analysis of the adversarial model and the low-dimensional latent feature representation to initialize the population of sparrow individuals, where each sparrow individual represents a hyperparameter combination. S32. Based on the hyperparameter combination corresponding to each individual sparrow, configure an analysis adversarial model and train it on a high-dimensional feature dataset to obtain performance evaluation indicators of the analysis adversarial model corresponding to each individual sparrow; S33, ranking the fitness of sparrow individuals according to the performance evaluation index of the analysis adversarial model, and dividing them into foragers and guards; S34. By simulating the three-stage action of sparrow foraging behavior, a position update strategy is implemented on foragers, a random perturbation operation is performed on guarders, and an escape migration operation is performed on some marginal individuals to generate a new generation of sparrow individual groups. Compared with the sparrow search algorithm, the improved sparrow search algorithm introduces an escape direction adjustment strategy based on global distribution feedback. The escape migration direction of marginal sparrow individuals has clustering guidance, which reduces the offset loss caused by random escape: in, represents the hyperparameter position vector of the i-th sparrow individual at the t-th iteration, represents the hyperparameter position vector of the i-th sparrow individual after update at the t+1th iteration, α represents the global search step adjustment factor, N represents the total number of sparrow individuals in the sparrow population, represents the hyperparameter vector position of the j-th sparrow individual at the t-th iteration, ||·|| represents the Euclidean distance, ∈ represents a small constant to prevent the denominator from being zero, and w j represents the fitness weight of the j-th sparrow individual, i represents the number of the sparrow individual whose position is currently being updated, j represents the reference individual number used to guide the update of the i-th sparrow individual, and t represents the number of the current search iteration round; S35. Continue configuring the hyperparameter combination of the new generation of sparrow individuals into the analysis and adversarial model, and repeat the training and evaluation process to form a global search mechanism based on the sparrow foraging behavior until the total number of iterations reaches 100 rounds; S36. Select sparrow individuals with a reconstruction error less than 0.015 from the final iterative sparrow individual population, and update the analysis adversarial model based on the hyperparameter combination corresponding to the sparrow individuals with a reconstruction error less than 0.

015.

7. The real-time analysis and fault location system for big data according to claim 2, characterized in that: The S4 includes the following specific steps: S41. Deploy the analysis adversarial model to the model running environment of the big data processing platform; S42, receiving a continuously input data stream in the analysis adversarial model running environment, and converting the data stream into a standardized high-dimensional feature data set according to a structured preprocessing rule; S43. Input the standardized high-dimensional feature dataset into the encoder of the deployed analytical adversarial model to generate a real-time low-dimensional latent feature representation sequence: in, represents the low-dimensional potential feature representation of the nth sample at time step τ, k represents the layer number of the encoder, n represents the data sample number, τ represents the time step number, Represents the high-dimensional feature vector of the nth sample at time step τ, Indicates the optimal model parameter combination The k-th layer encoder weight matrix is determined by Indicates the optimal model parameter combination The bias vector determined by K represents the total number of encoder modules, i represents the individual number of the sparrow whose position is currently being updated, and t represents the number of the current search iteration round; S44, inputting the low-dimensional latent feature representation into the decoder, generating real-time reconstructed samples, and calculating a new reconstruction error based on the input data and the reconstructed samples; S45. Stack the continuously generated low-dimensional latent feature representations in time series, extract the changing trends of features over time, and construct latent feature trajectories; S46. Jointly output the low-dimensional latent feature representation, the new reconstruction error, and the latent feature trajectory.

8. The big data-oriented real-time analysis and fault location system according to claim 2, characterized in that: The S5 includes the following specific steps: S51. Based on the low-dimensional latent feature representation and latent feature trajectory, the trajectory evolution sequence is constructed in chronological order; S52, constructing multiple sets of overlapping sliding windows based on the trajectory evolution sequence, and synchronously extracting trajectory change direction and amplitude information within each set of windows; S53, identifying the trajectory direction mutation behavior and the unstable fluctuation of feature expression in each overlapping window, and performing consistency check on the window intersection results to screen the candidate segments of the real trajectory change; S54, performing weighted fusion on the offset path length, offset duration, and change density of the trajectory in the candidate segment of the real trajectory change to construct an anomaly priority scoring matrix; S55. A dynamic judgment threshold is set based on the abnormal priority scoring matrix to determine whether a real-time abnormal event is constituted, and all trajectory segments that are below the threshold but appear continuously are stored in the abnormal memory vector pool: in, represents the abnormal score of the nth sample at time step τ, represents the k-th dimension component in the low-dimensional latent feature representation of the n-th sample at time step τ, D represents the total number of dimensions of the low-dimensional latent feature vector, W represents the length of the lookback window, w represents the historical time offset step within the window, n represents the data sample number, τ represents the time step number, and d represents the dimension number of the d-th dimension component in the latent feature vector; S56. Output the confirmed real-time abnormal event as a system-level resource exhaustion fault abnormal trajectory, and perform comparison and analysis on the trajectory pattern in the abnormal memory vector pool to output the final abnormal trajectory.

9. The real-time analysis and fault location system for big data according to claim 2, characterized in that: The S6 comprises the following specific steps: S61. Combining the final abnormal trajectory with the corresponding low-dimensional latent feature representation, constructing an abnormal event feature set; S62, representing a template of feature trajectories in the historical fault sample set and corresponding low-dimensional latent features; S63, performing structural alignment on the trajectory pattern sequence in the abnormal event feature set and the characteristic trajectory in the historical fault sample to generate a trajectory similarity matrix; S64, performing vector comparison analysis on the low-dimensional potential feature expression sequence in the abnormal event feature set and the feature trajectory in the historical sample to generate a feature representation similarity matrix; S65. Perform weighted fusion of the trajectory similarity matrix and the feature representation similarity matrix to construct a multi-dimensional matching score matrix. Calculate the weighted scores of each historical fault sample and the abnormal event feature in the trajectory similarity and feature vector similarity dimensions, with weight ratios of 0.6 and 0.

4. Take the historical sample with the highest total score of the weighted scores of the trajectory similarity and feature vector similarity dimensions and a score of not less than 0.85 as the most matching historical fault sample. S66. Extract the fault time index and source channel from the most matching historical fault sample and mark them as the positioning result of the current abnormal event feature.

10. The big data-oriented real-time analysis and fault location system according to claim 2, characterized in that: The S7 includes the following specific steps: S71. Extract the latest low-dimensional potential feature representation within the positioning window based on the fault time index and source channel, and matching with the timestamp and channel identifier in the current input data stream; S72. The latest low-dimensional potential feature representation is combined with the corresponding time index and source channel information to form the latest joint feature vector, and similarity matching is performed with the abnormal event features marked in the historical fault sample library. If the similarity is higher than 0.88 and the time index of the current abnormal event is offset from the time of the historical fault sample by no more than 3 time steps, it is determined to be the fault type corresponding to the historical sample, and the corresponding fault type label is output; S73: Bind the fault type label with the time index and the source channel to form a structured annotation result and output it to the label recording unit; S74. Perform a cross-check on the final abnormal trajectory and the current fault type label, and extract samples with high consistency as update data; S75. Input the updated data into the deployed analysis adversarial model to generate an updated analysis adversarial model parameter set; S76. Perform a stability evaluation on the updated analysis adversarial model parameter set. If the preset performance standard is met, replace the parameter configuration of the current analysis adversarial model to complete the online adaptive update process of the analysis adversarial model.

Images