Cloud platform fault intelligent diagnosis system and method based on multi-source data association analysis

Abstract

The invention discloses a cloud platform fault intelligent diagnosis system and method based on multi-source data association analysis, and belongs to the technical field of cloud computing. The system comprises a data acquisition module configured to be used for acquiring multi-source operation and maintenance data of a cloud platform in real time; the analysis processing module is configured to be used for preprocessing the collected multi-source operation and maintenance data and outputting an accurate fault result; the task distribution module is configured to be used for disassembling the fault diagnosis result into executable tasks and classifying the executable tasks according to fault types and emergency levels; and the operation and maintenance collaboration module is configured to be used for monitoring the execution progress of the executable task, coordinating all the operation and maintenance modules to work collaboratively and feeding back an execution result. According to the invention, accurate diagnosis, automatic positioning and efficient cooperative processing of cloud platform faults are realized, and the operation and maintenance efficiency and stability of the cloud platform are improved.

Description

Technical Field

[0001] This invention relates to a cloud platform fault intelligent diagnosis system and method based on multi-source data correlation analysis, belonging to the field of cloud computing technology. Background Technology

[0002] Currently, in the field of cloud platform operation and maintenance, cloud platforms have become the core platform for enterprises to deploy business systems and store core data. Their scale is constantly expanding and their architecture is becoming increasingly complex, encompassing multiple levels of components such as servers, network devices, storage devices, and upper-layer applications. The stable operation of cloud platforms directly determines the continuity of enterprise business; therefore, the timeliness and accuracy of fault diagnosis have become critical requirements in this field. With the trend of cloud platform operation and maintenance shifting from traditional manual operation and maintenance to automated operation and maintenance, the field faces challenges in efficiency and accuracy: On the one hand, cloud platform faults have the characteristics of "chain reaction" and "hidden nature." A single fault may trigger multi-dimensional alarms, and there are complex correlations between the root cause and the symptoms, increasing the difficulty of fault location; on the other hand, the configuration management data, monitoring alarm data, log data, and other multi-source information generated during operation and maintenance are independent of each other, making it difficult to form effective linkages, resulting in low fault location efficiency and poor diagnostic accuracy, failing to meet the high stability operation and maintenance requirements of cloud platforms.

[0003] Traditional cloud platform fault diagnosis methods mainly include single-data-dimensional diagnosis methods and human experience-driven diagnosis methods, and their specific implementation methods are as follows: 1) Single-data-dimensional diagnostic method: This method relies solely on a single type of operational data for fault diagnosis, such as locating faults only through monitoring and alarm data. When a cloud platform experiences a sudden spike in CPU utilization, operations personnel may only investigate the server process that generated the alarm based on the alarm information. However, they may not be able to further analyze the server's configuration change records (such as whether resource allocation has been recently adjusted) and system logs (such as whether there are abnormal process exit logs). This can easily lead to misjudging "CPU overload caused by insufficient resource allocation" as "abnormal CPU usage by processes," resulting in inaccurate fault location.

[0004] 2) Manual Experience-Driven Diagnostic Method: This method relies on the historical experience of senior operations and maintenance (O&M) personnel for fault diagnosis. When a cloud platform experiences a service response timeout, O&M personnel, based on their experience handling similar issues in the past, sequentially check network links, application servers, databases, and other related components. This requires manually querying independent data from each component (such as network logs, application logs, and database monitoring data) to verify hypotheses one by one. This process lacks a standardized procedure, and diagnostic efficiency depends entirely on personnel experience. Furthermore, for novel faults (with no historical experience to refer to), repeated trial and error are often necessary, resulting in a long fault handling cycle.

[0005] 3) Fault handling and collaboration: After locating the fault using the above two methods, the operation and maintenance personnel need to manually synchronize the fault information to the relevant execution links (such as notifying the resource configuration team to adjust server parameters, notifying the development team to fix application vulnerabilities). The teams communicate through instant messaging tools or emails, which lacks an automated collaboration mechanism and is prone to information transmission delays and omissions, further prolonging the fault recovery time.

[0006] The fundamental reason why existing technologies have the above problems is that: 1) Data fragmentation leads to low diagnostic accuracy: Existing technologies only utilize single-dimensional data for fault diagnosis, failing to establish correlations between multi-source data such as configuration management, monitoring alarms, and logs. For example, when a server experiences a memory leak alarm, without combining the server's application deployment configuration (such as whether new application instances have been added) and memory usage logs (such as whether the leak start time coincides with the application deployment time), it is impossible to accurately determine the root cause of the leak, easily leading to "band-aid" fault handling (such as only temporarily alleviating the problem by restarting the server, without resolving the application memory leak issue).

[0007] 2) Reliance on manual labor leads to low diagnostic efficiency: Traditional methods rely on the experience of operations and maintenance personnel and lack automated diagnostic capabilities. On the one hand, there is no unified standard for experience-based judgment, and different personnel have different troubleshooting paths for the same fault, which easily leads to duplication of work; on the other hand, in the face of the increasingly complex architecture and new types of faults of cloud platforms, human experience cannot cover all scenarios, resulting in repeated trial and error in fault location, with the mean time to locate faults (MTTD) reaching several hours, which is far from meeting the cloud platform's requirement of "minute-level fault response".

[0008] 3) Lack of collaboration leads to slow fault recovery: Existing technologies lack automated operation and maintenance collaboration mechanisms. After fault location, manual information synchronization and coordination of execution steps are required. For example, after locating "insufficient database connection pool configuration causing service timeout," operation and maintenance personnel need to manually notify the DBA (database administrator) to adjust the connection pool parameters. During this process, the DBA may not check the information in a timely manner, or the parameter adjustment may not be synchronized and fed back, resulting in delayed fault recovery and further expanding the scope of business impact. Summary of the Invention

[0009] The purpose of this invention is to overcome the shortcomings of the prior art and provide a cloud platform fault intelligent diagnosis system and method based on multi-source data correlation analysis. This invention solves the problems of low fault diagnosis accuracy, low efficiency, and slow fault recovery caused by data fragmentation, manual dependence, and lack of collaboration in the prior art. This enables accurate diagnosis, automatic location, and efficient collaborative processing of cloud platform faults, thereby improving the operation and maintenance efficiency and stability of the cloud platform. To achieve the above objectives / to solve the above technical problems, the present invention is implemented using the following technical solution: The first aspect: A cloud platform fault intelligent diagnosis system based on multi-source data correlation analysis, comprising: The data acquisition module is configured to collect multi-source operation and maintenance data from the cloud platform in real time. The analysis and processing module is configured to preprocess the collected multi-source operation and maintenance data and output accurate fault results. The task allocation module is configured to break down the fault diagnosis results into executable tasks and classify the executable tasks according to the fault type and urgency level. The operations and maintenance collaboration module is configured to monitor the execution progress of executable tasks, coordinate the collaborative work of various operations and maintenance modules, and provide feedback on the execution results.

[0010] Optionally, the data acquisition module includes: The configuration management data acquisition unit connects to the configuration management database of the cloud platform to periodically collect server configuration, application deployment configuration, and network topology configuration data, and adds timestamps and device identifiers to the collected configuration data and stores it in the module's built-in distributed data warehouse. The monitoring and alarm data acquisition unit collects CPU utilization, memory usage, disk IOPS, and service response time monitoring indicators in real time. When the indicators exceed the preset threshold, it automatically triggers alarms and collects alarm information. The log data acquisition unit collects logs from various levels using log acquisition tools, performs structured processing on the log data, removes invalid characters and duplicate logs, establishes a preliminary association with configuration management data and monitoring alarm data, and transmits the data to the analysis and processing module.

[0011] Optionally, the analysis and processing module includes: The data preprocessing unit receives multi-source data transmitted from the data acquisition module, first cleans the data, then standardizes the format, and finally removes redundant data. The processed data is then stored in the module's analysis database. The correlation analysis model unit, based on preprocessed multi-source data, constructs a "configuration-alarm-log" correlation rule base. The mined rules are sorted by confidence level, and rules with a confidence level > 80% are included in the correlation rule base for preliminary judgment of fault correlation data. The fault diagnosis engine builds a fault feature-type mapping library based on historical fault data. It integrates association matching rules for abnormal values ​​of monitoring indicators, alarm types, abnormal keywords in logs, and configuration change types. It adopts a multi-dimensional feature weighted scoring algorithm to establish the correspondence between feature combinations and fault types. When a new fault occurs, it matches the features extracted from real-time multi-source data with the rules in the mapping library, calculates the feature matching degree score, and performs cross-validation with high-confidence rules in the association rule library to output the fault diagnosis result.

[0012] Optionally, the task allocation module includes: The task parsing unit receives the fault diagnosis results pushed by the analysis and processing module, and then breaks down the specific tasks according to the fault type-processing task mapping relationship. The task scheduling unit allocates tasks to the corresponding operation and maintenance modules based on task type and priority.

[0013] Optionally, the operation and maintenance collaboration module includes: The task execution monitoring unit obtains the task execution status in real time by connecting to the API interfaces of various operation and maintenance modules; The result feedback unit collects fault recovery verification data after the task is completed and transmits it to the analysis and processing module to verify whether the fault has been truly recovered. If the fault has been recovered, a fault processing report is generated; if the fault has not been recovered, the analysis and processing module is triggered to re-diagnose.

[0014] Optionally, the data acquisition module and the analysis and processing module interact through an HTTP RESTful interface, transmitting processed multi-source data in batches every 30 seconds.

[0015] The second aspect: A cloud platform fault intelligent diagnosis method based on multi-source data correlation analysis, the method comprising: Step 1: Collaboratively collect multi-source operation and maintenance data. Step 2: Preprocess the collaboratively collected multi-source operation and maintenance data; Step 3: Match the preprocessed multi-source operation and maintenance data with the pre-built association rule base. If a high-confidence rule is matched, the core data corresponding to the rule is filtered out. If no existing rule is matched, the full amount of preprocessed data is retained. Step 4: Extract feature vectors from the core data or the full dataset, input the extracted feature vectors into the trained random forest fault classification model, and output preliminary diagnostic results; Step 5: Validate the preliminary diagnosis results based on the association rule base and output the final fault diagnosis results.

[0016] Optionally, the preprocessing of the collaboratively collected multi-source operation and maintenance data includes: Data cleaning: The 3σ principle is used to remove outliers from monitoring data and filter out blank lines, garbled characters, and invalid information from logs; Standardize the format: unify the units of monitoring indicators, unify the log timestamp format, and enumerate the configuration change types as 1 for new, 2 for modification, and 3 for deletion; Redundancy removal: Based on the triple key value of device ID + data type + timestamp, duplicate data collection and duplicate alarms caused by the same fault are deleted.

[0017] Optionally, the step of extracting feature vectors from core data or full data, inputting the extracted feature vectors into a trained random forest fault classification model, and outputting preliminary diagnostic results includes: Extract outliers, alarm types, log error keywords, and configuration change types from core data or full data, and quantify them; The quantified features are substituted into the multi-dimensional feature weighted scoring system of the fault diagnosis engine, and the diagnostic results are combined with the pre-built association rule base.

[0018] Optionally, the step of validating the preliminary diagnostic results based on the association rule base and outputting the final fault diagnosis results includes: If the preliminary diagnosis result matches the matching association rule by ≥90% and the diagnosis confidence is ≥85%, then the diagnosis result is confirmed to be valid. If the initial diagnosis results match the association rules by less than 90%, then supplement the missing data, re-extract the features, and input them into the random forest fault classification model for secondary inference to correct the diagnosis results. If no association rule is matched, the diagnostic confidence level will be used: if the confidence level is ≥85%, the result will be output directly; if the confidence level is <85%, it will be marked as requiring manual review.

[0019] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: 1. Improve fault diagnosis accuracy: By integrating configuration, monitoring, and log data through multi-source data collaborative collection and correlation mechanism, the limitations of a single data dimension are avoided; combined with the two-stage diagnostic model of "association rules + machine learning", the fault location deviation rate is reduced to below 5%, which is far lower than the deviation rate of more than 30% of traditional methods, thus achieving accurate fault diagnosis.

[0020] 2. Reduce fault location time: Through machine learning-based automatic diagnosis and standardized processes, the mean time to locate faults (MTTD) has been reduced from 4 hours using traditional manual methods to less than 10 minutes, significantly improving fault diagnosis efficiency, saving time for rapid fault recovery, and reducing business interruption losses.

[0021] 3. Accelerate fault recovery speed: Through an automated task allocation and collaborative processing system, fault handling does not require manual information synchronization. The response time of each operation and maintenance module is shortened to less than 5 minutes, and the mean time to recovery (MTTR) is shortened from 8 hours in the traditional method to less than 1 hour, significantly improving the stability of the cloud platform.

[0022] 4. Reduce maintenance manpower costs: Eliminate reliance on the experience of senior maintenance personnel. Junior maintenance personnel can complete fault handling based on the diagnostic results and task guidance output by the system, reducing manpower costs by more than 40%. At the same time, automated processes reduce repetitive work, allowing maintenance personnel to devote their energy to higher-value work such as optimizing cloud platform architecture. Attached Figure Description

[0023] Figure 1 The diagram shown is a schematic of the cloud platform fault intelligent diagnosis system of the present invention; Figure 2 The diagram shown is a schematic of the cloud platform fault intelligent diagnosis method of the present invention. Detailed Implementation

[0024] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0025] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are used only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0026] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0027] Example 1, such as Figure 1 A cloud platform fault intelligent diagnosis system based on multi-source data correlation analysis is disclosed, comprising: The data acquisition module is configured to collect multi-source operation and maintenance data from the cloud platform in real time. The analysis and processing module is configured to preprocess the collected multi-source operation and maintenance data and output accurate fault results. The task allocation module is configured to break down the fault diagnosis results into executable tasks and classify the executable tasks according to the fault type and urgency level. The operations and maintenance collaboration module is configured to monitor the execution progress of executable tasks, coordinate the collaborative work of various operations and maintenance modules, and provide feedback on the execution results.

[0028] In this embodiment, the data acquisition module's function is to comprehensively and in real-time collect multi-source operation and maintenance data from the cloud platform, providing a data foundation for subsequent analysis and diagnosis. Internally, it includes a configuration management data acquisition unit, a monitoring and alarm data acquisition unit, and a log data acquisition unit. Configuration Management Data Acquisition Unit: By connecting to the configuration management database (CMDB) of the cloud platform, it periodically (every 5 minutes by default) collects static and dynamic configuration data such as server configuration (number of CPU cores, memory capacity, storage path), application deployment configuration (number of application instances, deployment nodes, dependent service addresses), and network topology configuration (switch port mapping, routing rules). It adds timestamps and device identifiers to the collected configuration data to ensure data traceability and stores it in the module's built-in distributed data warehouse.

[0029] Monitoring and alarm data acquisition unit: Deploy agent programs to various servers and network devices to collect monitoring indicators such as CPU utilization, memory utilization, disk IOPS (input / output operations per second), and service response time in real time; when the indicators exceed preset thresholds (such as CPU utilization > 90%, response time > 500ms), an alarm is automatically triggered and alarm information (alarm type, occurrence time, associated device ID) is collected; monitoring data and alarm data are stored in association according to the "device-time" dimension, and a monitoring snapshot is updated every 10 seconds.

[0030] Log data acquisition unit: Collects logs at various levels through log acquisition tools (such as Fluentd), including operating system logs ( / var / log / messages), application logs (such as log4j logs for Java applications), and database logs (such as binlog for MySQL); performs structured processing on the log data (extracting log level, timestamp, event description, and associated process ID), removes invalid characters and duplicate logs, establishes a preliminary association with configuration management data and monitoring alarm data through "device ID + time range", and transmits it to the analysis and processing module.

[0031] This module interacts with the analysis and processing module via an HTTP RESTful interface, transmitting processed multi-source data in batches every 30 seconds to ensure that the analysis and processing module obtains the latest and most complete operation and maintenance data.

[0032] In this embodiment, the analysis and processing module performs preprocessing, correlation analysis, and machine learning diagnosis on multi-source data, outputting accurate fault types and root causes. Internally, it includes a data preprocessing unit, a correlation analysis model unit, and a machine learning diagnosis unit. Data preprocessing unit: After receiving multi-source data from the data acquisition module, it first performs data cleaning (removing outliers from monitoring data, such as "CPU utilization = -1%" caused by Agent failure; filtering invalid information in logs, such as empty log lines); then it performs format standardization (unifying the units of monitoring indicators for different devices, such as converting memory usage to percentages; unifying log timestamps to the "YYYY-MM-DDHH:MM:SS" format); finally, it removes redundant data (deleting duplicate configuration change records and duplicate alarms caused by the same fault), and stores the processed data in the module's analysis database.

[0033] The association analysis model unit constructs a "configuration-alarm-log" association rule base based on preprocessed multi-source data. Specifically, it employs association rule mining algorithms (such as the Apriori algorithm) to analyze the co-occurrence relationships of "configuration change events," "alarm events," and "log anomaly events" in historical data. For example, it mines an association rule for "within 2 hours after the server memory configuration was adjusted from 16GB to 8GB, an alarm with 'memory utilization > 95%' appeared, and the system log contained 'out of memory' records." The mined rules are then sorted by confidence level (the probability of the rule being true), and rules with a confidence level > 80% are included in the association rule base for preliminary judgment of fault-related data. Its core function is to provide "data linkage basis" for fault diagnosis, narrowing the diagnostic scope, eliminating irrelevant interference, and verifying the rationality of machine learning model results. The fault diagnosis engine builds a fault feature-type mapping library based on historical fault data. It integrates association matching rules for abnormal values ​​of monitoring indicators, alarm types, abnormal keywords in logs, and configuration change types. It adopts a multi-dimensional feature weighted scoring algorithm (the weight of each feature is marked by historical data, such as 0.4 for abnormal monitoring indicators, 0.3 for log keywords, 0.2 for alarm types, and 0.1 for configuration change types) to establish the correspondence between feature combinations and fault types. When a new fault occurs, the features extracted from real-time multi-source data are matched with the rules in the mapping library to calculate the feature matching degree score. Combined with cross-validation of high-confidence rules in the association rule library, the fault diagnosis result is output.

[0034] The training process of the random forest model is as follows: 10,800 valid historical fault samples from the cloud platform over the past 3 years were selected (divided into training, validation, and test sets in a 7:2:1 ratio). After labeling the fault type and root cause, 11 core features, including outliers in monitoring indicators, alarm weights, log keyword frequency, and the impact of configuration changes, were extracted and standardized using Z-Score. Parallel training was performed with parameters such as 100 decision trees and a maximum depth of 15 layers. The convergence was monitored in real time using the validation set. The SMOTE algorithm was used to balance the samples and select high-importance features to optimize the model. The final model achieved an accuracy of 92.3% on the test set. After serialized deployment, incremental training and updates were supported, and the inference time after inputting new fault data was ≤500ms.

[0035] This module interacts with the task allocation module via a message queue (such as RabbitMQ). Once the fault diagnosis results are output, they are immediately pushed to the task allocation module to ensure that fault handling starts quickly.

[0036] In this embodiment, the task allocation module functions to break down the fault diagnosis results into executable tasks and allocate them to the corresponding operation and maintenance modules. Internally, it includes a task parsing unit and a task scheduling unit. Task parsing unit: After receiving the fault diagnosis results pushed by the analysis and processing module, it breaks down the specific tasks according to the "fault type-processing task" mapping relationship. For example, if the diagnosis result is "insufficient resource configuration (low memory configuration)", it is broken down into the task of "adjusting the target server's memory configuration to 16GB"; if the diagnosis result is "application vulnerability (Java application memory leak)", it is broken down into the task of "restarting the application service + notifying the development team to analyze the memory leak code"; each task must clearly define the task content, the execution target (such as the specific server IP, development team identifier), and the execution time limit (such as 30 minutes for emergency fault tasks and 2 hours for general fault tasks).

[0037] Task scheduling unit: Based on task type and priority (urgent fault tasks are set to "high", general faults to "medium", and potential fault warnings to "low"), tasks are assigned to the corresponding operation and maintenance modules. For example, the "adjust memory configuration" task is assigned to the resource configuration module, the "restart application service" task is assigned to the application operation and maintenance module, and the "notify the development team" task is assigned to the collaborative office module. At the same time, through task scheduling algorithms (such as the shortest job first algorithm), high-priority tasks with short execution times are executed first. After task assignment, the task status (pending execution, in execution, completed, failed) is tracked in real time and synchronized to the operation and maintenance collaboration module.

[0038] This module interacts synchronously with the operations and maintenance collaboration module through a database. Task allocation information is written to the shared database in real time, and the operations and maintenance collaboration module can read task details and status at any time.

[0039] In this embodiment, the operation and maintenance collaboration module functions to monitor task execution progress, coordinate the collaborative work of various operation and maintenance modules, and provide feedback on execution results. Internally, it includes a task execution monitoring unit and a result feedback unit. Task Execution Monitoring Unit: By connecting to the API interfaces of various operation and maintenance modules, it obtains the task execution status in real time. For example, by connecting to the resource configuration module API, it queries whether the "adjust memory configuration" task has been completed; by connecting to the application operation and maintenance module API, it queries the execution progress of the "restart application service" task; if a task execution times out (e.g., a high-priority task has not been completed within 30 minutes), it automatically sends a reminder to the corresponding module manager (via SMS or WeChat Work); if a task execution fails (e.g., memory configuration adjustment fails due to permission issues), it generates a failure reason analysis (e.g., "The current account does not have permission to modify server hardware configuration") and feeds it back to the task allocation module.

[0040] Results Feedback Unit: After the task is completed, it collects fault recovery verification data (such as "memory utilization has recovered to 60%" and service response time has returned to the normal range) from the monitoring module, and transmits it to the analysis and processing module to verify whether the fault has been truly recovered. If the fault has been recovered, a fault handling report is generated (including the fault diagnosis process, the executed task, and the recovery time). If the fault has not been recovered (such as memory leaks still occurring after the application is restarted), the analysis and processing module is triggered to re-diagnose, forming a closed loop of "diagnosis-processing-verification-re-diagnosis".

[0041] This module interacts with the analysis and processing module via an HTTP interface. Task execution results and fault recovery data are pushed to the analysis and processing module in real time, ensuring closed-loop management of fault diagnosis and handling.

[0042] like Figure 2 As shown, a cloud platform fault intelligent diagnosis method based on multi-source data correlation analysis is disclosed, the method comprising: Collect configuration management data, monitoring warning data, and log data; The collected configuration management data, monitoring warning data, and log data are cleaned, standardized, and redundant data are removed. Association rules are applied to the preprocessed data to construct a rule base with a confidence level greater than 80%. Real-time multi-source data is input into the trained random forest model, and combined with rule base verification, the fault diagnosis results are output. Based on the fault type-processing task mapping relationship, the fault diagnosis results are decomposed into task priorities and types, and the execution objects and time limits are clearly defined. If the fault is resolved, a processing report is generated; if it is not resolved, the analysis and processing module is triggered to re-diagnose.

[0043] For detailed methods, please refer to [link / reference]. Figure 2To further elaborate: This method includes the following steps: Step 1, Collaborative Collection of Multi-Source Operation and Maintenance Data: The data acquisition module collects cloud platform operation and maintenance data through its three acquisition units, forming a complete data input: Configuration Management Data Collection: Connects to the cloud platform configuration management database (CMDB) to periodically collect server configuration (number of CPU cores, memory capacity), application deployment configuration (number of instances, deployment nodes), and network topology configuration (switch port mapping) data, and adds timestamps and device identifiers; Monitoring and alarm data collection: Real-time collection of monitoring metrics such as CPU utilization, memory usage, disk IOPS, and service response time through the Agent. When the metrics exceed the preset threshold, alarms are triggered and information such as alarm type and outlier values ​​are collected. Log data collection: Collect operating system, application, and database level logs using log collection tools, perform structured processing (extract log levels and keywords), remove invalid characters and duplicate logs, and establish preliminary correlations with configuration and monitoring data.

[0044] Step 2: Multi-source data preprocessing: The data preprocessing unit of the analysis and processing module receives all the data collected in step 1 and processes it according to the following process: Data cleaning: Use the 3σ principle to remove outliers from monitoring data (such as CPU utilization = -1%) and filter out invalid information such as blank lines and garbled characters in logs; Standardize the format: unify the units of monitoring indicators (e.g., convert memory usage to %), unify the log timestamps to the format "YYYY-MM-DDHH:MM:SS", and assign values ​​for configuration change types by enumerating "Add=1, Modify=2, Delete=3". Redundancy removal: Based on the triple key value of "device ID + data type + timestamp", duplicate data and duplicate alarms caused by the same fault are deleted, and the processed data is stored in the analysis database.

[0045] Step 3: Matching and filtering by association rules: The correlation analysis model unit of the analysis and processing module calls the pre-built "configuration-alarm-log" correlation rule base (confidence > 80%) to match the data preprocessed in step 2 with the rule base: If a high-confidence rule is matched (such as "memory configuration downgrade → memory utilization over-limit alarm → log contains OOM keyword"), then the core data corresponding to the rule (such as configuration change records, memory alarm information, and OOM logs for this fault) will be selected as the key input for subsequent model diagnosis. If no existing rule is matched (e.g., a new type of fault), the preprocessed full data is retained and proceeds to the model diagnosis stage.

[0046] Step 4: Engine Diagnosis: Fault Diagnosis The fault diagnosis engine in the analysis and processing module performs precise diagnosis. The specific process is as follows: Feature extraction: From the core data (or full data) filtered in step 3, extract four key features and perform quantification: Anomalies in monitoring metrics: Calculate the percentage of CPU utilization exceeding the threshold (90%) (anomaly = actual value - 90%), the cumulative duration of memory utilization consistently above 95%, the multiple by which disk IOPS exceeds the threshold, and the duration of service response time exceeding 500ms, forming quantitative characteristic values; Alarm type: Assign values ​​according to preset impact weights (hardware alarm = 0.8, application alarm = 0.5, network alarm = 0.6), and calculate the sum of the weights of all alarm types associated with this fault as the result of alarm feature quantification; Log anomaly keywords: Count the frequency of occurrence of preset high-frequency fault words such as "OOM", "connection refused", and "resource limit exceeded". A frequency ≥ 1 is counted as a valid feature, and the cumulative frequency is used as the log feature quantification result; Configuration change type: Assign values ​​according to the degree of impact (hardware configuration change = 0.9, software configuration change = 0.6, network configuration change = 0.7), mark the configuration change records within 72 hours before this failure, and take the highest weight value of the corresponding change type as the configuration feature quantification result.

[0047] Rule matching and scoring reasoning: The quantified four types of features are substituted into the multi-dimensional feature weighted scoring system of the fault diagnosis engine, and matching reasoning is performed in conjunction with a pre-built "feature combination - fault type" mapping library (based on 10,000+ historical fault data points). Weighted scoring calculation: The total feature matching score (out of 100 points) is calculated according to the preset ratio of "0.4 for abnormal values ​​of monitoring indicators, 0.3 for abnormal keywords in logs, 0.2 for alarm types, and 0.1 for configuration change types". Fault type matching: Based on the total score, match the fault type threshold range in the mapping library (e.g., insufficient resource configuration corresponds to a score of 80-100 points, application vulnerability corresponds to a score of 60-79 points, etc.) to lock in candidate fault types; Fault root cause localization: Based on the candidate fault type, further match the corresponding rules of "feature combination and root cause" under that type (e.g., "monitoring indicator abnormal value = memory over-limit + configuration change type = hardware configuration change" corresponds to the root cause "memory configuration too low"). Diagnostic confidence calculation: The confidence level is the ratio of the total feature score to the full score of the threshold range for that fault type (e.g., a score of 85 corresponds to an insufficient resource allocation type, so the confidence level is 85%). The final output is the preliminary diagnostic result, including: Fault type (e.g., insufficient resource allocation, application vulnerability, network failure, hardware damage); Root causes of the problem (such as insufficient memory configuration, Java application memory leak, switch port failure). Diagnostic confidence (the probability of reliability corresponding to the rule matching score).

[0048] Step 5: Validate and correct association rules: The correlation analysis model unit of the analysis and processing module verifies the preliminary diagnostic results of step 4: If the preliminary diagnosis result is highly consistent with the association rule matched in step 3 (matching degree ≥ 90%) and the diagnosis confidence is ≥ 85%, then the diagnosis result is confirmed to be valid. If the initial diagnostic results match the association rules by less than 90% (e.g., the model determines "application vulnerability" but the rule base has no corresponding log-alarm association), then supplement the missing data (e.g., application thread stack logs, dependent service status data), re-extract the features and input them into the model for secondary reasoning to correct the diagnostic results; If no matching rule is found (new type of fault), the diagnostic confidence level is used as a reference: if the confidence level is ≥85%, the result is output directly; if the confidence level is <85%, it is marked as "requires manual review" and pushed to the operation and maintenance collaboration module.

[0049] Step 6: Output the final fault diagnosis results: After verification and correction in step 5, a complete fault diagnosis result is output, including "fault type, fault root cause, diagnostic confidence level, related data links (such as specific information of configuration change → alarm → log), and suggested handling measures." This result is pushed to the task allocation module for subsequent task breakdown and execution. If the diagnosis result is marked as "requires manual review," it is synchronized to the operation and maintenance collaboration module for supplementary confirmation by operation and maintenance personnel before proceeding to the task allocation stage. The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A cloud platform fault intelligent diagnosis system based on multi-source data correlation analysis, characterized in that, include: The data acquisition module is configured to collect multi-source operation and maintenance data from the cloud platform in real time. The analysis and processing module is configured to preprocess the collected multi-source operation and maintenance data and output accurate fault results. The task assignment module is configured to break down the fault diagnosis results into executable tasks and classify the executable tasks according to the fault type and urgency level. The operations and maintenance collaboration module is configured to monitor the execution progress of executable tasks, coordinate the collaborative work of various operations and maintenance modules, and provide feedback on the execution results.

2. The cloud platform fault intelligent diagnosis system based on multi-source data correlation analysis according to claim 1, characterized in that, The data acquisition module includes: The configuration management data acquisition unit connects to the configuration management database of the cloud platform to periodically collect server configuration, application deployment configuration, and network topology configuration data, and adds timestamps and device identifiers to the collected configuration data and stores it in the module's built-in distributed data warehouse. The monitoring and alarm data acquisition unit collects CPU utilization, memory usage, disk IOPS, and service response time monitoring indicators in real time. When the indicators exceed the preset threshold, it automatically triggers alarms and collects alarm information. The log data acquisition unit collects logs from various levels using log acquisition tools, performs structured processing on the log data, removes invalid characters and duplicate logs, establishes a preliminary association with configuration management data and monitoring alarm data, and transmits the data to the analysis and processing module.

3. The cloud platform fault intelligent diagnosis system based on multi-source data correlation analysis according to claim 1, characterized in that, The analysis and processing module includes: The data preprocessing unit receives multi-source data transmitted from the data acquisition module, first cleans the data, then standardizes the format, and finally removes redundant data. The processed data is then stored in the module's analysis database. The correlation analysis model unit, based on preprocessed multi-source data, constructs a "configuration-alarm-log" correlation rule base. The mined rules are sorted by confidence level, and rules with a confidence level >80% are included in the correlation rule base for preliminary judgment of fault correlation data. The fault diagnosis engine builds a fault feature-type mapping library based on historical fault data. It integrates association matching rules for abnormal values ​​of monitoring indicators, alarm types, abnormal keywords in logs, and configuration change types. It adopts a multi-dimensional feature weighted scoring algorithm to establish the correspondence between feature combinations and fault types. When a new fault occurs, it matches the features extracted from real-time multi-source data with the rules in the mapping library, calculates the feature matching degree score, and performs cross-validation with high-confidence rules in the association rule library to output the fault diagnosis result.

4. The cloud platform fault intelligent diagnosis system based on multi-source data correlation analysis according to claim 1, characterized in that, The task allocation module includes: The task parsing unit receives the fault diagnosis results pushed by the analysis and processing module, and then breaks down the specific tasks according to the fault type-processing task mapping relationship. The task scheduling unit allocates tasks to the corresponding operation and maintenance modules based on task type and priority.

5. The cloud platform fault intelligent diagnosis system based on multi-source data correlation analysis according to claim 1, characterized in that, The operation and maintenance collaboration module includes: The task execution monitoring unit obtains the task execution status in real time by connecting to the API interfaces of various operation and maintenance modules; The result feedback unit collects fault recovery verification data after the task is completed and transmits it to the analysis and processing module to verify whether the fault has been truly recovered. If the fault has been recovered, a fault processing report is generated; if the fault has not been recovered, the analysis and processing module is triggered to re-diagnose.

6. The cloud platform fault intelligent diagnosis system based on multi-source data correlation analysis according to claim 1, characterized in that, The data acquisition module and the analysis and processing module interact through an HTTP RESTful interface, transmitting processed multi-source data in batches every 30 seconds.

7. A cloud platform fault intelligent diagnosis method based on multi-source data correlation analysis, characterized in that, The method includes: Step 1: Collaboratively collect multi-source operation and maintenance data. Step 2: Preprocess the collaboratively collected multi-source operation and maintenance data; Step 3: Match the preprocessed multi-source operation and maintenance data with the pre-built association rule base. If a high-confidence rule is matched, the core data corresponding to the rule is filtered out. If no existing rule is matched, the full amount of preprocessed data is retained. Step 4: Extract feature vectors from the core data or the full dataset, input the extracted feature vectors into the trained random forest fault classification model, and output preliminary diagnostic results; Step 5: Validate the preliminary diagnosis results based on the association rule base and output the final fault diagnosis results.

8. The cloud platform fault intelligent diagnosis method based on multi-source data correlation analysis according to claim 7, characterized in that, The preprocessing of collaboratively collected multi-source operation and maintenance data includes: Data cleaning: The 3σ principle is used to remove outliers from monitoring data and filter out blank lines, garbled characters, and invalid information from logs; Standardize the format: unify the units of monitoring indicators, unify the log timestamp format, and enumerate the configuration change types as 1 for new, 2 for modification, and 3 for deletion; Redundancy removal: Based on the triple key value of device ID + data type + timestamp, duplicate data collection and duplicate alarms caused by the same fault are deleted.

9. The cloud platform fault intelligent diagnosis method based on multi-source data correlation analysis according to claim 7, characterized in that, The process involves extracting feature vectors from core or full data, inputting these feature vectors into a trained random forest fault classification model, and outputting preliminary diagnostic results, including: Extract outliers, alarm types, log error keywords, and configuration change types from core data or full data, and quantify them; The quantified features are substituted into the multi-dimensional feature weighted scoring system of the fault diagnosis engine, and the diagnostic results are combined with the pre-built association rule base.

10. The cloud platform fault intelligent diagnosis method based on multi-source data correlation analysis according to claim 7, characterized in that, The preliminary diagnostic results are validated based on the association rule base, and the final fault diagnosis results are output, including: If the preliminary diagnosis result matches the matching association rule by ≥90% and the diagnosis confidence is ≥85%, then the diagnosis result is confirmed to be valid. If the initial diagnosis results match the association rules by less than 90%, then supplement the missing data, re-extract the features, and input them into the random forest fault classification model for secondary inference to correct the diagnosis results. If no association rule is matched, the diagnostic confidence level will be used: if the confidence level is ≥85%, the result will be output directly; if the confidence level is <85%, it will be marked as requiring manual review.

Images