Cross-platform data security processing and intelligent operation and maintenance management system based on multi-cloud service

Abstract

The invention provides a cross-platform data security processing and intelligent operation and maintenance management system based on multi-cloud service, and belongs to the field of intelligent operation and maintenance management, and the system comprises a distributed data collection module which is used for obtaining performance monitoring data and problem report data from a plurality of cloud service platforms; the data processing module is used for constructing a mapping relation between different cloud service platforms and performing field conversion and format standardization processing; the dual anomaly detection module comprises a performance index anomaly detection sub-module for analyzing data anomaly fluctuation and a problem mode recognition sub-module for analyzing an event anomaly growth mode; and the operation and maintenance management module is used for performing association analysis based on the data exception fluctuation and the event exception growth mode, determining an exception processing priority and triggering an automatic processing flow of the corresponding cloud service platform. According to the method, the automation degree of cross-platform operation and maintenance and the accuracy of exception handling are improved, and full-process intelligent management from data acquisition, standardized processing, exception detection to automatic operation and maintenance is realized.

Description

Technical Field

[0001] This invention relates to the field of intelligent operation and maintenance management technology, and in particular to a cross-platform data security processing and intelligent operation and maintenance management system based on multi-cloud services. Background Technology

[0002] With the rapid development of cloud computing technology and the deepening of enterprise digital transformation, more and more enterprises are adopting multi-cloud strategies to meet the needs of different business scenarios, achieving flexibility and reliability in their business architecture by integrating the resources and services of multiple cloud service providers. A multi-cloud environment not only avoids over-reliance on a single cloud service provider and reduces vendor lock-in risk, but also improves overall system stability through resource complementarity and redundant backups. However, the complexity of multi-cloud environments also brings unprecedented challenges to enterprise operation and maintenance management.

[0003] Existing multi-cloud operation and maintenance technologies suffer from the following technical shortcomings: First, data heterogeneity is a serious problem, with different cloud service platforms employing their own independent data formats, interface specifications, and monitoring indicator systems, making cross-platform data integration and analysis difficult. Second, most existing intelligent operation and maintenance systems are designed for single-cloud environments and lack the ability to handle complex data relationships in multi-cloud scenarios, resulting in insufficient accuracy and response delays in anomaly detection and fault diagnosis. Third, traditional operation and maintenance management tools rely heavily on manual configuration and maintenance when standardizing cross-platform data, leading to low automation and a high risk of errors. Finally, existing technologies lack the ability to perform anomaly correlation analysis in multi-cloud environments, making it difficult to accurately identify cascading faults and complex anomaly patterns across platforms.

[0004] Chinese invention patent CN118585372A discloses a cloud service management system based on intelligent operation and maintenance of multi-cloud applications. This system includes a multi-cloud access module, an intelligent operation and maintenance module, an automatic scaling module, and a cost optimization module. It achieves unified management of the multi-cloud environment through a fault detection unit, a performance optimization unit, and a resource scheduling unit. It uses machine learning algorithms to analyze logs and monitoring data for fault detection, adjusts resources based on preset performance indicators, and supports intelligent resource scheduling across regions. However, it fails to solve the fundamental problem of data heterogeneity across different cloud service platforms, leading to a decrease in the accuracy of operation and maintenance management. Summary of the Invention

[0005] In view of this, the present invention proposes a cross-platform data security processing and intelligent operation and maintenance management system based on multi-cloud services, which can solve the problem of cross-platform data heterogeneity in the prior art and improve the automation level of cross-platform operation and maintenance and the accuracy of anomaly handling.

[0006] The technical solution of this invention is implemented as follows: This invention provides a cross-platform data security processing and intelligent operation and maintenance management system based on multi-cloud services, comprising:

[0007] The distributed data acquisition module is used to obtain performance monitoring data and problem report data from multiple cloud service platforms respectively;

[0008] The data processing module is used to construct mapping relationships between different cloud service platforms, and to perform field conversion and format standardization processing on performance monitoring data and problem report data based on the mapping relationships between different cloud service platforms to obtain standardized data; the standardized data includes standardized performance monitoring data and standardized problem report data.

[0009] The dual anomaly detection module includes a performance indicator anomaly detection submodule and a problem pattern recognition submodule. The performance indicator anomaly detection submodule is used to analyze abnormal fluctuations in standardized performance monitoring data, and the problem pattern recognition submodule is used to analyze abnormal growth patterns of events in standardized problem report data.

[0010] The operations and maintenance management module is used to perform correlation analysis based on abnormal data fluctuations and abnormal event growth patterns, determine the priority of anomaly handling, and trigger the corresponding automated processing flow of the cloud service platform.

[0011] Based on the above technical solutions, preferably, the distributed data acquisition module adopts a three-layer architecture design, including:

[0012] The data source access layer is used to access the application interfaces of various cloud service platforms through a lightweight monitoring data collection client, and to collect performance monitoring data and problem report data from different cloud service platforms.

[0013] The data transmission processing layer is used to queue and buffer performance monitoring data and problem report data from different cloud service platforms according to the first-in-first-out principle, and then perform data compression, data encryption and data routing and distribution processing in sequence.

[0014] The data aggregation and storage layer adopts a distributed storage architecture to store performance monitoring data and problem report data separately. The distributed storage architecture includes a data sharding storage mechanism, a multi-replica backup strategy, and a load balancing scheduling algorithm. Performance monitoring data is stored in a time-series database, and problem report data is stored in a document database.

[0015] Based on the above technical solutions, preferably, the steps of the distributed data acquisition module specifically include:

[0016] The lightweight monitoring data acquisition client sets tiered acquisition frequencies based on the frequency of changes in performance indicators.

[0017] The lightweight monitoring data acquisition client collects alarm events, service failure reports, and error logs through the event notification interface of the cloud service platform, and extracts the cloud service platform identifier, service type, geographical region, and timestamp to form structured problem report data;

[0018] The data transmission processing layer establishes an encrypted transmission channel between the lightweight monitoring data acquisition client and the data aggregation and storage layer. The acquired performance monitoring data and problem report data are encrypted using the transport layer security protocol before being transmitted to the data aggregation and storage layer.

[0019] The data aggregation and storage layer receives and decrypts encrypted data, routes performance monitoring data to the time series database based on the data type identifier field, and routes problem report data to the document database.

[0020] Based on the above technical solutions, preferably, the data processing module specifically includes:

[0021] The metadata management unit is used to store data source connection information, field definitions and data type definitions, and conversion mapping rules between fields for each cloud service platform using a relational database.

[0022] The intelligent mapping engine is used to read field definitions and conversion mapping rules between fields from the metadata management unit, and to build mapping relationships between different cloud service platforms through a standardized dictionary. The standardized dictionary includes a unified naming convention and corresponding relationship table for service type, geographical region, and resource type.

[0023] The data processing unit is used to perform field name conversion, data type format unification, and timestamp standardization on performance monitoring data and problem report data according to the field mapping relationship provided by the intelligent mapping engine, so as to obtain standardized data, and send the data processing process and data processing records to the metadata management unit.

[0024] Based on the above technical solutions, preferably, the step of constructing the mapping relationship between different cloud service platforms through a standardized dictionary specifically includes:

[0025] Obtain the application programming interface (API) documents and data structure specifications of various cloud service platforms, and automatically identify the correspondence between fields across different platforms;

[0026] The correspondence between fields across different platforms is semantically matched and verified with the standardized terms in the dictionary. The accuracy of the term mapping is determined by calculating the similarity of word vectors, resulting in a bidirectional mapping table between the terms of each cloud service platform and the standard terms.

[0027] Store the bidirectional mapping table between the terminology of each cloud service platform and the standard terminology in the mapping rule base.

[0028] Based on the above technical solutions, the preferred method for the performance index anomaly detection submodule is as follows:

[0029] Standardized performance monitoring data is preprocessed to obtain complete time-series performance data;

[0030] The mean μ and standard deviation σ of each performance indicator are calculated based on historical performance data, and a threshold range of 3 times [μ-3σ, μ+3σ] is established.

[0031] A time series prediction model based on the Transformer architecture is constructed. Historical performance data is divided into input sequences according to time windows. The multi-head self-attention mechanism is used to learn the dependence and periodic patterns of performance indicators in the time dimension. The time series prediction model is trained to predict the normal numerical range of each time point.

[0032] The currently monitored performance data points are compared with both the statistical threshold range and the Transformer model prediction range. If the currently monitored performance data points exceed the statistical threshold range and the asymmetric deviation ratio exceeds the preset threshold, the data points are judged to be abnormal.

[0033] An anomaly severity score is calculated based on the magnitude of the deviation of the abnormal data points from the normal range and the duration of the abnormal state. The anomaly severity score is used as the result of numerical anomaly fluctuation.

[0034] Based on the above technical solutions, preferably, the formula for calculating the asymmetric deviation ratio is as follows:

[0035]

[0036] δ=(x current -x predicted ) / |x predicted |;

[0037]

[0038] Where R represents the asymmetric deviation ratio, x current x represents the currently monitored performance data point. predicted δ represents the center value of the prediction interval of the Transformer model, f(δ) represents the deviation direction, f(δ) represents the asymmetric sensitivity function, and k represents the sensitivity adjustment parameter.

[0039] Based on the above technical solutions, the preferred steps of the problem pattern recognition submodule are as follows:

[0040] Standardized issue report data are grouped and statistically analyzed according to the combination of cloud service platform identifier, service type identifier and geographic region code, and a time series of issue events for each combination is established.

[0041] A fixed time window is used to segment and statistically analyze the time series of problem events for each combination, calculate the number of problem events occurring within each time window, and establish trend data of the number of problem events changing over time.

[0042] The sliding time window technique is used to perform growth analysis on the trend data of the number of issues. When the number of issues in the most recent time window exceeds twice the historical average for the same period, the combination is marked as an abnormal growth state.

[0043] Set a threshold for the number of issues on the cloud service platform, and filter each combination based on the issue threshold to obtain combinations with abnormal data growth;

[0044] The data combinations with abnormal growth are sorted in descending order based on the number of issues, and the sorting results are used as the outcome of the abnormal growth pattern of events.

[0045] Based on the above technical solutions, preferably, the correlation analysis based on abnormal data fluctuations and abnormal event growth patterns specifically includes:

[0046] Establish a unified abnormal event data structure, which includes an anomaly type identifier, occurrence timestamp, scope of impact on the cloud service platform, severity score, and detection confidence.

[0047] When abnormal data fluctuations and abnormal event growth patterns occur simultaneously within a preset time window and within the same cloud service platform, the temporal correlation coefficient and spatial overlap of the two anomalies are calculated. When the absolute value of the temporal correlation coefficient is greater than the temporal correlation threshold and the spatial overlap is greater than the spatial overlap threshold, it is determined that the abnormal data fluctuations and abnormal event growth patterns originate from the same fundamental problem.

[0048] A priority calculation model is established to calculate a priority score based on the number of cloud service platforms affected by the anomaly, the duration of the anomaly, and the level of business importance. The anomaly events are then sorted according to the priority scores to obtain an anomaly event sequence.

[0049] More preferably, the automated processing flow of the operation and maintenance management module specifically includes:

[0050] When abnormal data fluctuations are received, automated operation and maintenance operations are triggered based on a predefined rule engine. These automated operation and maintenance operations include automatic capacity expansion, load redistribution, and fault isolation.

[0051] When the result of the abnormal growth pattern of the event is received, a special analysis process is initiated, which includes problem confirmation, impact scope assessment, expert team notification, and in-depth investigation.

[0052] When an anomaly in the correlation is confirmed by correlation analysis, the special analysis process is executed first, and the abnormal data fluctuation information is input into the special analysis process as supplementary data.

[0053] Select the notification channel and the scope of recipients based on the severity score of the anomaly.

[0054] The cross-platform data security processing and intelligent operation and maintenance management system based on multi-cloud services of the present invention has the following advantages over the prior art:

[0055] (1) By adopting a distributed data acquisition module to acquire performance monitoring data and problem report data of multi-cloud service platforms, a unified cross-platform data mapping relationship is established and standardized conversion is completed through a data processing module. A dual anomaly detection mechanism is used to identify abnormal fluctuations in values ​​and abnormal growth patterns of events respectively. Correlation analysis is performed through the operation and maintenance management module and corresponding automated processing is triggered. This significantly improves the automation level of cross-platform operation and maintenance and the accuracy of anomaly handling, and realizes intelligent management of the entire process from data acquisition, standardized processing, anomaly detection to automated operation and maintenance.

[0056] (2) Through the collaborative work of the metadata management unit, intelligent mapping engine and data processing unit, a complete cross-platform data standardization processing mechanism has been established, which has improved the efficiency of data quality management and problem investigation.

[0057] (3) By using the performance index anomaly detection submodule and the problem pattern recognition submodule to process performance monitoring data and problem report data respectively, dual verification of performance anomalies is achieved, and risks can be detected in a timely manner. The results are fused by unifying the abnormal event data structure. When anomalies are detected at the same time and in the same range, correlation analysis is automatically performed, which effectively avoids treating different manifestations of the same problem as independent events. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 This is a block diagram of the cross-platform data security processing and intelligent operation and maintenance management system based on multi-cloud services of the present invention;

[0060] Figure 2 This is a schematic block diagram of the distributed data acquisition module of the cross-platform data security processing and intelligent operation and maintenance management system based on multi-cloud services of the present invention.

[0061] Figure 3 This is a schematic block diagram of the dual anomaly detection module of the cross-platform data security processing and intelligent operation and maintenance management system based on multi-cloud services of the present invention. Detailed Implementation

[0062] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0063] like Figure 1 As shown, this invention provides a cross-platform data security processing and intelligent operation and maintenance management system based on multi-cloud services, including:

[0064] The distributed data acquisition module is used to obtain performance monitoring data and problem report data from multiple cloud service platforms. The performance monitoring data includes system and application-level metrics such as CPU utilization, memory consumption, disk I / O, network bandwidth, response time, throughput, and error rate. The problem report data includes alarm events, service failure reports, error logs, and abnormal event records.

[0065] The data processing module is used to construct mapping relationships between different cloud service platforms, and to perform field conversion and format standardization processing on performance monitoring data and problem report data based on the mapping relationships between different cloud service platforms to obtain standardized data; the standardized data includes standardized performance monitoring data and standardized problem report data.

[0066] The dual anomaly detection module includes a performance indicator anomaly detection submodule and a problem pattern recognition submodule. The performance indicator anomaly detection submodule is used to analyze abnormal fluctuations in standardized performance monitoring data, and the problem pattern recognition submodule is used to analyze abnormal growth patterns of events in standardized problem report data.

[0067] The operations and maintenance management module is used to perform correlation analysis based on abnormal data fluctuations and abnormal event growth patterns, determine the priority of anomaly handling, and trigger the corresponding automated processing flow of the cloud service platform.

[0068] Understandably, performance monitoring data is continuous event sequence data used to reflect the system's operating status, while problem report data is discrete event data used to record system anomalies. Abnormal fluctuations in performance monitoring data are often quantitative manifestations of fault events recorded in problem report data. For example, when CPU utilization suddenly spikes (abnormal performance monitoring data), it may correspond to a service anomaly recorded in the error log (problem report data).

[0069] This invention employs a distributed data acquisition module to obtain performance monitoring data and problem report data from multi-cloud service platforms. A data processing module establishes a unified cross-platform data mapping relationship and completes standardized conversion. A dual anomaly detection mechanism identifies abnormal numerical fluctuations and abnormal event growth patterns. An operations and maintenance management module performs correlation analysis and triggers corresponding automated processing flows. This effectively solves the management complexity problem caused by data heterogeneity in multi-cloud environments, significantly improves the automation level of cross-platform operations and maintenance and the accuracy of anomaly handling, and achieves intelligent management of the entire process from data acquisition, standardized processing, anomaly detection, and automated operations and maintenance.

[0070] like Figure 2 As shown, in one embodiment of the present invention, the distributed data acquisition module adopts a three-layer architecture design, including:

[0071] The data source access layer is used to access the application interfaces of various cloud service platforms through a lightweight monitoring data collection client, and to collect performance monitoring data and problem report data from different cloud service platforms.

[0072] The data transmission processing layer is used to queue and buffer performance monitoring data and problem report data from different cloud service platforms according to the first-in-first-out principle, and then perform data compression, data encryption and data routing and distribution processing in sequence.

[0073] The data aggregation and storage layer adopts a distributed storage architecture to store performance monitoring data and problem report data separately. The distributed storage architecture includes a data sharding storage mechanism, a multi-replica backup strategy, and a load balancing scheduling algorithm. Performance monitoring data is stored in a time-series database, and problem report data is stored in a document database.

[0074] Understandably, lightweight monitoring data acquisition clients call the standard monitoring API interfaces of various cloud service platforms via HTTP / HTTPS protocols, obtain event data through webhook mechanisms or API polling, and then extract cloud service platform identifiers (such as AWS, Azure, GCP), service type identifiers (such as compute, storage, network), geographic region codes (such as us-east-1, asia-southeast1), and timestamp information, and organize this information into a structured problem report data format.

[0075] The queue buffer sends performance monitoring data and issue reports from different cloud service platforms to the Kafka Producer. The Producer then distributes these data to the corresponding Topic partitions based on their data type identifiers. Upon receiving the data, the Kafka Broker stores it in a distributed log in chronological order.

[0076] This invention achieves standardization and automation of multi-cloud data collection by unifying access to application programming interfaces (APIs) of different cloud service platforms through a lightweight monitoring data collection client. At the same time, the data transmission processing layer uses message queues for buffering, which solves the complexity and consistency problems of data collection in multi-cloud environments and ensures the reliability and security of data transmission.

[0077] In one embodiment of the present invention, data compression employs the LZ4 compression algorithm, and data encryption employs...

[0078] The AES-256-GCM symmetric encryption algorithm uses a key managed centrally by a Key Management Service (KMS) and rotated periodically. Data routing and distribution are based on the type identifier field in the message header, routing performance monitoring data to the time-series database processing channel and issue report data to the document database processing channel.

[0079] In one embodiment of the present invention, the data sharding storage mechanism performs hash sharding based on timestamps and cloud service platform identifiers to ensure that data is evenly distributed across storage nodes.

[0080] In one embodiment of the present invention, the multi-replica backup strategy adopts a 3-replica configuration, with the primary replica responsible for read and write operations and the secondary replicas performing asynchronous synchronization to ensure data reliability.

[0081] In one embodiment of the present invention, the load balancing scheduling algorithm adopts the consistent hashing algorithm, which supports dynamic scaling of storage nodes.

[0082] In one embodiment of the present invention, the steps of the distributed data acquisition module specifically include:

[0083] The lightweight monitoring data acquisition client sets tiered acquisition frequencies based on the frequency of changes in performance indicators.

[0084] The lightweight monitoring data acquisition client collects alarm events, service failure reports, and error logs through the event notification interface of the cloud service platform, and extracts the cloud service platform identifier, service type, geographical region, and timestamp to form structured problem report data;

[0085] The data transmission processing layer establishes an encrypted transmission channel between the lightweight monitoring data acquisition client and the data aggregation and storage layer. The acquired performance monitoring data and problem report data are encrypted using the transport layer security protocol before being transmitted to the data aggregation and storage layer.

[0086] The data aggregation and storage layer receives and decrypts encrypted data, routes performance monitoring data to the time series database based on the data type identifier field, and routes problem report data to the document database.

[0087] Specifically, an end-to-end encrypted transmission channel is established between the data acquisition client and the data aggregation and storage layer using the TLS 1.3 protocol.

[0088] After receiving encrypted data, the data aggregation and storage layer first performs TLS decryption, and then determines the route based on the data type identifier field in the message header. For data with the type identifier "performance_metric", the route is stored in the corresponding measurement in the time series database; for data with the type identifier "incident_report", the route is stored in the corresponding index in the document database.

[0089] In one embodiment of the present invention, the data processing module specifically includes:

[0090] The metadata management unit is used to store data source connection information, field definitions and data type definitions, and conversion mapping rules between fields for each cloud service platform using a relational database.

[0091] The intelligent mapping engine is used to read field definitions and conversion mapping rules between fields from the metadata management unit, and to build mapping relationships between different cloud service platforms through a standardized dictionary. The standardized dictionary includes a unified naming convention and corresponding relationship table for service type, geographical region, and resource type.

[0092] The data processing unit is used to perform field name conversion, data type format unification, and timestamp standardization on performance monitoring data and problem report data according to the field mapping relationship provided by the intelligent mapping engine, so as to obtain standardized data, and send the data processing process and data processing records to the metadata management unit.

[0093] Understandably, the metadata management unit uses a PostgreSQL relational database to store data source connection information, field definitions and data type definitions, and conversion mapping rules between fields for each cloud service platform. The database design adopts a star schema, including four core tables: data source table (data_sources), field definition table (field_definitions), mapping rule table (mapping_rules), and transformation history table (transformation_history).

[0094] The system includes several key components: a data source connection information table recording the API access point URL, authentication method, access key, and connection parameters for each cloud service platform; a field definition table recording the field names, data types, value ranges, unit information, and semantic descriptions for each cloud service platform; a mapping rule table recording the correspondence between fields across different platforms, conversion functions, and validation rules; and a conversion history table recording detailed logs of the data processing process, supporting data lineage tracing and problem troubleshooting. When a new cloud service platform is added, new mapping rules can be added via a web interface or API call, automatically validating the syntax correctness and logical consistency of the rules.

[0095] This invention establishes a complete cross-platform data standardization processing mechanism through the collaborative work of a metadata management unit, an intelligent mapping engine, and a data processing unit. This effectively solves the problem of inconsistent data formats in multi-cloud environments and improves the efficiency of data quality management and problem troubleshooting.

[0096] In one embodiment of the present invention, field name conversion is based on a mapping rule table. Each field of the input data is traversed, and the corresponding standard field name is searched in the mapping rule table to complete the field renaming operation. For fields where no mapping rule is found, an optimal matching algorithm is used to attempt to recommend possible mapping relationships, and these are recorded in an error log for manual handling.

[0097] Understandably, the data processing unit sends the data processing flow and results records to the metadata management unit to achieve full-process tracking and quality monitoring of data processing. The processing records include the input data source, the mapping rules of the application, the comparison of the data structure before and after the transformation, the processing time, and abnormal information.

[0098] Furthermore, data type format standardization includes numeric type conversion, string encoding standardization, and Boolean value standardization. Timestamp standardization converts all time data to the ISO 8601 standard format in the UTC time zone.

[0099] In one embodiment of the present invention, constructing a mapping relationship between different cloud service platforms through a standardized dictionary specifically includes:

[0100] Obtain the application programming interface (API) documents and data structure specifications of various cloud service platforms, and automatically identify the correspondence between fields across different platforms;

[0101] The correspondence between fields across different platforms is semantically matched and verified with the standardized terms in the dictionary. The accuracy of the term mapping is determined by calculating the similarity of word vectors, resulting in a bidirectional mapping table between the terms of each cloud service platform and the standard terms.

[0102] Store the bidirectional mapping table between the terminology of each cloud service platform and the standard terminology in the mapping rule base.

[0103] Specifically, when the word vector similarity exceeds 0.8, the term mapping relationship is automatically determined; when the word vector similarity is between 0.5 and 0.8, it is submitted for manual review; and when the word vector similarity is below 0.5, the mapping between cloud service platform terms and standard terms is rejected.

[0104] like Figure 3 As shown, in one embodiment of the present invention, the specific steps of the performance index anomaly detection submodule are as follows:

[0105] Standardized performance monitoring data is preprocessed to obtain complete time-series performance data;

[0106] The mean μ and standard deviation σ of each performance indicator are calculated based on historical performance data, and a threshold range of 3 times [μ-3σ, μ+3σ] is established.

[0107] A time series prediction model based on the Transformer architecture is constructed. Historical performance data is divided into input sequences according to time windows. The multi-head self-attention mechanism is used to learn the dependence and periodic patterns of performance indicators in the time dimension. The time series prediction model is trained to predict the normal numerical range of each time point.

[0108] The currently monitored performance data point is subjected to dual evaluation against both the statistical threshold range and the Transformer model prediction range. If the currently monitored performance data point simultaneously exceeds the statistical threshold range and the asymmetric deviation ratio exceeds a preset threshold, the data point is determined to be abnormal. The calculation formula is as follows:

[0109]

[0110] δ=(x current -x predicted ) / |x predicted |;

[0111]

[0112] Where R represents the asymmetric deviation ratio, x currentx represents the currently monitored performance data point. predicted δ represents the center value of the prediction interval of the Transformer model, f(δ) represents the deviation direction, f(δ) represents the asymmetric sensitivity function, and k represents the sensitivity adjustment parameter.

[0113] An anomaly severity score is calculated based on the magnitude of the deviation of outlier data points from the normal range and the duration of the anomaly. This anomaly severity score is taken as the result of numerical anomaly fluctuations. The calculation formula is as follows:

[0114]

[0115] Among them, S severity The score represents the severity of the anomaly, α represents the weighting coefficient of the deviation magnitude, and D... magnitude T represents the deviation magnitude, β represents the weighting coefficient for duration, and T represents the deviation magnitude. duration Indicates duration.

[0116] Understandably, the deviation ratio calculation formula compares the currently monitored performance data points with the normal values ​​predicted by the Transformer model. This fully considers the significant differences in performance baselines among different cloud service platforms in a multi-cloud environment. By using a relative ratio rather than an absolute value for anomaly detection, it avoids false alarms caused by platform differences. In cross-platform data processing scenarios across multiple cloud services, the performance baselines of different cloud service providers vary significantly. Through the calculation of the relative deviation ratio, the system can adapt to these platform differences, achieving fairer and more accurate anomaly detection.

[0117] The formula for calculating the severity score of anomalies uses a logarithmic function to handle the magnitude of deviation and a square root function to handle the duration. The logarithmic function can compress the influence of extreme values ​​and prevent individual extreme outliers from dominating the overall score. The square root function reflects the diminishing marginal effect of the time factor, that is, after the duration of the anomaly increases to a certain extent, its marginal contribution to the severity gradually decreases.

[0118] This invention employs a multi-head self-attention mechanism to simultaneously capture short-term fluctuation patterns (minute-level performance jitter), medium-term trends (hour-level load changes), and long-term cycles (daily, weekly, and monthly business cycles). Although the complete time-series performance data has undergone standardization, it still retains the periodic characteristics of each platform. The Transformer's self-attention mechanism can identify the different patterns of the platform during Monday morning peak hours and Friday evening peak hours. Location encoding can accurately capture the behavioral characteristics of different cloud platform services at different points in time. At the same time, it improves prediction efficiency by processing the time series of multiple cloud platforms simultaneously through parallel computing.

[0119] Specifically, preprocessing includes data cleaning, missing value imputation, and noise removal. Data cleaning employs outlier identification methods based on statistical regularities, using box plot analysis to identify obvious outliers, and marking and processing extreme values ​​exceeding 1.5 times the interquartile range. Missing value imputation uses time series interpolation methods; linear interpolation is used for short-term missing values ​​(less than 5 time points), while historical averages are used for long-term missing values. Noise removal uses a moving average filter to smooth the data, with the window size dynamically adjusted according to the acquisition frequency.

[0120] Furthermore, the Transformer architecture employs an Encoder-Decoder structure. The Encoder receives historical time series data as input and extracts temporal dependency features through a multi-layer self-attention mechanism. The Decoder predicts the numerical range of future time points based on the Encoder's output. Multi-head attention captures temporal dependencies across different dimensions by computing multiple attention heads in parallel.

[0121] MultiHead(Q,K,V)=Concat(head1,…,head h W O

[0122]

[0123] Where MultiHead(,K,V) represents the output of the multi-head attention mechanism, Q represents the query matrix, K represents the key matrix, V represents the value matrix, Concat(·) represents the tensor concatenation operation, head1 represents the output of the first attention head, and head... h Let W represent the output of the h-th attention head, b represent the total number of attention heads, and W represent the output of the h-th attention head. O Indicates the output linear transformation matrix, head j This represents the output of the j-th attention head, where j represents the attention head index. This describes the specific calculation process for the j-th attention head. This represents the query weight matrix corresponding to the j-th attention head. This represents the key weight matrix corresponding to the j-th attention head. This represents the value weight matrix corresponding to the j-th attention head.

[0124] In one embodiment of the present invention, the steps of the problem pattern recognition submodule are as follows:

[0125] Standardized issue report data are grouped and statistically analyzed according to the combination of cloud service platform identifier, service type identifier and geographic region code, and a time series of issue events for each combination is established.

[0126] A fixed time window is used to segment and statistically analyze the time series of problem events for each combination, calculate the number of problem events occurring within each time window, and establish trend data of the number of problem events changing over time.

[0127] A sliding time window technique is used to perform growth analysis on the trend data of the number of issues. When the number of issues in the most recent time window exceeds twice the historical average for the same period, the combination is marked as an abnormal growth state. The logic of the growth analysis is as follows:

[0128]

[0129] Among them, R growth Indicates the growth factor, Count recent Mean represents the number of issues in the most recent time window. historical This indicates the average number of questions during the same historical period;

[0130] Set a threshold for the number of issues on the cloud service platform, and filter each combination based on the issue threshold to obtain combinations with abnormal data growth;

[0131] The data combinations with abnormal growth are sorted in descending order based on the number of issues, and the sorting results are used as the outcome of the abnormal growth pattern of events.

[0132] Understandably, the cloud service platform identifier, service type identifier, and geographic region code form a combination, and each combination forms an independent issue counting time series with a time granularity of 1 hour, counting the total number of issue events that occur within that time period.

[0133] This invention achieves dual verification of performance anomalies by employing a performance index anomaly detection submodule and a problem pattern recognition submodule to process performance monitoring data and problem report data respectively, and can detect risks in a timely manner. It achieves result fusion by using a unified abnormal event data structure, and automatically performs correlation analysis when anomalies are detected simultaneously within similar time and related range, effectively avoiding the situation of treating different manifestations of the same problem as independent events.

[0134] The performance anomaly detection submodule can promptly identify potential problems when system performance parameters deviate from the normal range. By combining 3σ threshold judgment with the Transformer prediction model, it can provide early warnings in the early stages of performance degradation, avoiding false alarms that may occur when relying solely on threshold judgment. The problem pattern recognition submodule can capture service quality degradation that users have already perceived but which has not yet been reflected in performance monitoring indicators by analyzing the changing trends in the number of user problem reports.

[0135] In one embodiment of the present invention, the problem threshold for the number of problems on the cloud service platform is set to 5%, and only combinations that reach the problem threshold are considered as meaningful abnormal patterns.

[0136] In one embodiment of the present invention, correlation analysis based on abnormal data fluctuations and abnormal event growth patterns specifically includes:

[0137] Establish a unified abnormal event data structure, which includes an anomaly type identifier, occurrence timestamp, scope of impact on the cloud service platform, severity score, and detection confidence.

[0138] When abnormal data fluctuations and abnormal event growth patterns occur simultaneously within a preset time window and on the same cloud service platform, the temporal correlation coefficient and spatial overlap of the two anomalies are calculated. If the absolute value of the temporal correlation coefficient is greater than the temporal correlation threshold and the spatial overlap is greater than the spatial overlap threshold, then the abnormal data fluctuations and abnormal event growth patterns are determined to originate from the same fundamental problem. The calculation formula is as follows:

[0139]

[0140] Among them, R time The time correlation coefficient is represented by i, where i is the index of the time point, n is the total number of time points included in the analysis time window, and x is the index of the time point. i This represents the intensity of the numerical anomaly at time point i. y represents the average value of abnormal fluctuations in numerical values. i This indicates the abnormal intensity of the abnormal growth pattern of the event at time point i. O represents the average value of the abnormal growth pattern of events. space Let P1 represent the set of cloud service platforms affected by abnormal numerical fluctuations, P2 represent the set of cloud service platforms affected by abnormal event growth patterns, |P1∩P2| represent the number of cloud service platforms that exhibit both abnormal phenomena simultaneously, and |P1∪P2| represent the total number of cloud service platforms that exhibit either abnormal phenomenon.

[0141] A priority calculation model is established to calculate a priority score based on the number of cloud service platforms affected by the anomaly, the duration of the anomaly, and the level of business importance. The anomaly events are then sorted according to the priority scores to obtain an anomaly event sequence.

[0142] Understandably, aligning the two types of anomalous events by timestamps to establish a time series X = {x1, x2, ..., x...} n} and Y = {y1, y2, ..., y n}, where x i y represents the severity of numerical anomalies within the i-th time window. i This represents the number of event anomalies within the i-th time window.

[0143] The anomaly type identifier includes two types: abnormal data fluctuation and abnormal growth pattern. The occurrence timestamp is used to record the precise time when the anomaly was detected. The scope of cloud service platforms affected is used to record the list of cloud service platforms involved in the anomaly, including the cloud service platform name, service type and geographical region information. The severity score is based on the scope and duration of the impact and ranges from 0 to 100. The detection confidence score represents the reliability of the detection algorithm in judging the anomaly and ranges from 0 to 1.

[0144] In the calculation of the time correlation coefficient, for abnormal data fluctuations, the abnormal intensity can be defined as the degree to which the current data point deviates from the normal range; for abnormal event growth patterns, the abnormal intensity can be defined as the growth multiple of the number of problematic events at the current time point relative to the historical baseline.

[0145] In the calculation of spatial overlap, P1 and P2 should include not only the cloud service provider's identifier, but also the specific service type and geographical area information.

[0146] When data fluctuates abnormally, automated operation and maintenance (O&M) actions are triggered, such as automatic capacity expansion, load redistribution, and fault isolation. When events increase abnormally, an analysis process is triggered, such as problem confirmation, impact assessment, and notification to the expert team. If the abnormal data fluctuations and abnormal event growth patterns stem from the same underlying issue, they will not be handled separately but will be processed uniformly according to priority.

[0147] This invention achieves intelligent correlation analysis of different types of abnormal events by calculating time correlation coefficients and spatial overlap, and quantifies the priority score of anomaly handling through mathematical formulas, ensuring the reasonable allocation of operation and maintenance resources and timely response to high-priority issues, thereby improving the effectiveness of operation and maintenance decisions in a multi-cloud environment.

[0148] In one embodiment of the present invention, the method for calculating the priority score specifically includes:

[0149] Establish an evaluation index set H = {h1, h2, h3, h4, h5}, where h1 represents the severity of numerical anomalies, i.e., the proportion of asymmetric deviation; h2 represents the severity of event anomalies, i.e., the multiple by which event anomalies increase; h3 represents the spatiotemporal correlation strength; h4 represents the impact on platform complexity; and h5 represents the time urgency. h3 = |R time |×O space h5 = min(1, T / 60), where T is the duration;

[0150] Construct a pairwise comparison matrix A = (a pq )5×5, where a pq Indicator h p Relative to h q The importance of;

[0151] For m abnormal events and n = 5 evaluation indicators, construct the decision matrix D = (d pq ) m×5 ;

[0152] Calculate the information entropy of the q-th indicator:

[0153] Calculate the standardized probability value:

[0154] Calculate entropy weights:

[0155] W Combined =α·W AHP +(1-α)·W Entropy

[0156] Optimal α value:

[0157] Where, d pq E represents the raw value of the p-th anomalous event on the q-th evaluation metric. q Let represent the information entropy value of the q-th indicator, m represent the total number of outliers, p represent the index of the outlier, q represent the index of the evaluation indicator, and g represent the information entropy value of the q-th indicator. qp This represents the standardized probability value of the p-th anomalous event on the q-th indicator. W represents the objective weight of the q-th indicator calculated using the entropy weight method. Combined Let W represent the final combined weight vector, where α represents W. AHP This represents the subjective weight vector of AHP. This represents the subjective weight of the q-th indicator calculated using the AHP method, where P represents the priority score. This represents the final combined weight of the q-th indicator.

[0158] Understandably, each anomalous event has multiple evaluation metrics, such as the severity of numerical anomalies from Transformer, the severity of event anomalies from problem identification, the temporal and spatial correlation from correlation analysis, the platform complexity from intelligent mapping, and the time urgency. The AHP-entropy weight method can find the best balance point through optimization based on the objective characteristics of actual data, and thus accurately determine the priority score.

[0159] In one embodiment of the present invention, the automated processing flow of the operation and maintenance management module specifically includes:

[0160] When abnormal data fluctuations are received, automated operation and maintenance operations are triggered based on a predefined rule engine. These automated operation and maintenance operations include automatic capacity expansion, load redistribution, and fault isolation.

[0161] When the result of the abnormal growth pattern of the event is received, a special analysis process is initiated, which includes problem confirmation, impact scope assessment, expert team notification, and in-depth investigation.

[0162] When an anomaly in the correlation is confirmed by correlation analysis, the special analysis process is executed first, and the abnormal data fluctuation information is input into the special analysis process as supplementary data.

[0163] Based on the severity score of the anomaly, select the notification channels and the scope of recipients. Notification channels include email, SMS, instant messaging tools, and telephone.

[0164] Understandably, predefined rule engines include conditional rules based on exception type and severity:

[0165] CPU usage anomaly handling rules: When CPU usage exceeds 85% for more than 5 minutes, an automatic expansion operation is triggered; the expansion strategy adopts a step-by-step expansion, first increasing computing resources by 20%, and if the anomaly persists after 3 minutes, the expansion continues.

[0166] Network latency anomaly handling rules: When the network response time exceeds 200% of the normal baseline, a load redistribution operation is triggered, and the system automatically switches some traffic to a backup node or geographical area with lower latency;

[0167] Service response failure handling rules: When the service response failure rate exceeds 5%, the fault node isolation operation is triggered, and the system automatically removes the fault node from the load balancer pool to prevent the fault from spreading.

[0168] Specifically, the specialized analysis process includes: automatically analyzing the service components, user groups, and geographical areas involved in the anomaly, and generating an impact scope report; based on the service type and platform characteristics of the anomaly, the system automatically notifies relevant technical experts to ensure timely intervention of professional knowledge; the system provides the technical team with relevant historical data, performance trend charts, and timelines of related events to assist in root cause analysis; based on the analysis results, the system recommends possible solution templates and tracks the progress of the resolution.

[0169] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A cross-platform data security processing and intelligent operation and maintenance management system based on multi-cloud services, characterized in that: include: The distributed data acquisition module is used to obtain performance monitoring data and problem report data from multiple cloud service platforms respectively; The data processing module is used to construct mapping relationships between different cloud service platforms, and to perform field conversion and format standardization processing on performance monitoring data and problem report data based on the mapping relationships between different cloud service platforms to obtain standardized data; the standardized data includes standardized performance monitoring data and standardized problem report data. The dual anomaly detection module includes a performance indicator anomaly detection submodule and a problem pattern recognition submodule. The performance indicator anomaly detection submodule is used to analyze abnormal fluctuations in standardized performance monitoring data, and the problem pattern recognition submodule is used to analyze abnormal growth patterns of events in standardized problem report data. The operations and maintenance management module is used to perform correlation analysis based on abnormal data fluctuations and abnormal event growth patterns, determine the priority of anomaly handling, and trigger the corresponding automated processing flow of the cloud service platform.

2. The cross-platform data security processing and intelligent operation and maintenance management system based on multi-cloud services as described in claim 1, characterized in that: The distributed data acquisition module adopts a three-tier architecture design, including: The data source access layer is used to access the application interfaces of various cloud service platforms through a lightweight monitoring data collection client, and to collect performance monitoring data and problem report data from different cloud service platforms. The data transmission processing layer is used to queue and buffer performance monitoring data and problem report data from different cloud service platforms according to the first-in-first-out principle, and then perform data compression, data encryption and data routing and distribution processing in sequence. The data aggregation and storage layer adopts a distributed storage architecture to store performance monitoring data and problem report data separately. The distributed storage architecture includes a data sharding storage mechanism, a multi-replica backup strategy, and a load balancing scheduling algorithm. Performance monitoring data is stored in a time-series database, and problem report data is stored in a document database.

3. The cross-platform data security processing and intelligent operation and maintenance management system based on multi-cloud services as described in claim 1, characterized in that: The specific steps of the distributed data acquisition module include: The lightweight monitoring data acquisition client sets tiered acquisition frequencies based on the frequency of changes in performance indicators. The lightweight monitoring data acquisition client collects alarm events, service failure reports, and error logs through the event notification interface of the cloud service platform, and extracts the cloud service platform identifier, service type, geographical region, and timestamp to form structured problem report data; The data transmission processing layer establishes an encrypted transmission channel between the lightweight monitoring data acquisition client and the data aggregation and storage layer. The acquired performance monitoring data and problem report data are encrypted using the transport layer security protocol before being transmitted to the data aggregation and storage layer. The data aggregation and storage layer receives and decrypts encrypted data, routes and stores performance monitoring data to the time series database based on the data type identifier field, and routes and stores problem report data to the document database.

4. The cross-platform data security processing and intelligent operation and maintenance management system based on multi-cloud services as described in claim 1, characterized in that: The data processing module specifically includes: The metadata management unit is used to store data source connection information, field definitions and data type definitions, and conversion mapping rules between fields for each cloud service platform using a relational database. The intelligent mapping engine is used to read field definitions and conversion mapping rules between fields from the metadata management unit, and to build mapping relationships between different cloud service platforms through a standardized dictionary. The standardized dictionary includes a unified naming convention and corresponding relationship table for service type, geographical region, and resource type. The data processing unit is used to perform field name conversion, data type format unification, and timestamp standardization on performance monitoring data and problem report data according to the field mapping relationship provided by the intelligent mapping engine, so as to obtain standardized data, and send the data processing process and data processing records to the metadata management unit.

5. The cross-platform data security processing and intelligent operation and maintenance management system based on multi-cloud services as described in claim 4, characterized in that: The construction of mapping relationships between different cloud service platforms through a standardized dictionary specifically includes: Obtain the application programming interface (API) documents and data structure specifications of various cloud service platforms, and automatically identify the correspondence between fields across different platforms; The correspondence between fields across different platforms is semantically matched and verified with the standardized terms in the dictionary. The accuracy of the term mapping is determined by calculating the similarity of word vectors, resulting in a bidirectional mapping table between the terms of each cloud service platform and the standard terms. Store the bidirectional mapping table between the terminology of each cloud service platform and the standard terminology in the mapping rule base.

6. The cross-platform data security processing and intelligent operation and maintenance management system based on multi-cloud services as described in claim 1, characterized in that: The specific steps of the performance indicator anomaly detection submodule are as follows: Standardized performance monitoring data is preprocessed to obtain complete time-series performance data; The mean μ and standard deviation σ of each performance indicator are calculated based on historical performance data, and a threshold range of 3 times [μ-3σ, μ+3σ] is established. A time series prediction model based on the Transformer architecture is constructed. Historical performance data is divided into input sequences according to time windows. The multi-head self-attention mechanism is used to learn the dependence and periodic patterns of performance indicators in the time dimension. The time series prediction model is trained to predict the normal numerical range of each time point. The currently monitored performance data points are compared with both the statistical threshold range and the Transformer model prediction range. If the currently monitored performance data points exceed the statistical threshold range and the asymmetric deviation ratio exceeds the preset threshold, the data points are judged to be abnormal. An anomaly severity score is calculated based on the magnitude of the deviation of the abnormal data points from the normal range and the duration of the abnormal state. The anomaly severity score is used as the result of numerical anomaly fluctuation.

7. The cross-platform data security processing and intelligent operation and maintenance management system based on multi-cloud services as described in claim 6, wherein the calculation formula for the asymmetric deviation ratio is: δ=(x current -x predicted ) / |x predicted |; in, R represents the asymmetric deviation ratio, x current X represents the currently monitored performance data point. predicted δ represents the center value of the prediction interval of the Transformer model, f(δ) represents the deviation direction, f(δ) represents the asymmetric sensitivity function, and k represents the sensitivity adjustment parameter.

8. The cross-platform data security processing and intelligent operation and maintenance management system based on multi-cloud services as described in claim 7, characterized in that: The steps of the problem pattern recognition submodule are as follows: Standardized issue report data are grouped and statistically analyzed according to the combination of cloud service platform identifier, service type identifier and geographic region code, and a time series of issue events for each combination is established. A fixed time window is used to segment and statistically analyze the time series of problem events for each combination, calculate the number of problem events occurring within each time window, and establish trend data of the number of problem events changing over time. The sliding time window technique is used to perform growth analysis on the trend data of the number of issues. When the number of issues in the most recent time window exceeds twice the historical average for the same period, the combination is marked as an abnormal growth state. Set a threshold for the number of issues on the cloud service platform, and filter each combination based on the issue threshold to obtain combinations with abnormal data growth; The data combinations with abnormal growth are sorted in descending order based on the number of issues, and the sorting results are used as the outcome of the abnormal growth pattern of events.

9. The cross-platform data security processing and intelligent operation and maintenance management system based on multi-cloud services as described in claim 8, characterized in that: The correlation analysis based on abnormal data fluctuations and abnormal event growth patterns specifically includes: Establish a unified abnormal event data structure, which includes an anomaly type identifier, occurrence timestamp, scope of impact on the cloud service platform, severity score, and detection confidence. When abnormal data fluctuations and abnormal event growth patterns occur simultaneously within a preset time window and within the same cloud service platform, the temporal correlation coefficient and spatial overlap of the two anomalies are calculated. When the absolute value of the temporal correlation coefficient is greater than the temporal correlation threshold and the spatial overlap is greater than the spatial overlap threshold, it is determined that the abnormal data fluctuations and abnormal event growth patterns originate from the same fundamental problem. A priority calculation model is established to calculate a priority score based on the number of cloud service platforms affected by the anomaly, the duration of the anomaly, and the level of business importance. The anomaly events are then sorted according to the priority scores to obtain an anomaly event sequence.

10. The cross-platform data security processing and intelligent operation and maintenance management system based on multi-cloud services as described in claim 9, characterized in that: The automated processing flow of the operation and maintenance management module specifically includes: When abnormal data fluctuations are received, automated operation and maintenance operations are triggered based on a predefined rule engine. These automated operation and maintenance operations include automatic capacity expansion, load redistribution, and fault isolation. When the result of the abnormal growth pattern of the event is received, a special analysis process is initiated, which includes problem confirmation, impact scope assessment, expert team notification, and in-depth investigation. When an anomaly in the correlation is confirmed by correlation analysis, the special analysis process is executed first, and the abnormal data fluctuation information is input into the special analysis process as supplementary data. Select the notification channel and the scope of recipients based on the severity score of the anomaly.

Images