Big data driven network operation state monitoring management method and system

Abstract

The invention discloses a big-data-driven network operation state monitoring management method and system, and relates to the technical field of digital information monitoring, and the method comprises the steps: building a distributed database, storing multi-modal data from network equipment, link states, user behaviors and security logs, carrying out the time synchronization and semantic association of the multi-modal data, and storing the multi-modal data in the distributed database; constructing an abnormal state multi-dimensional association graph; a multi-level collaborative network is used for collaborative recognition, an additional abnormal result is generated by combining abnormal positioning and time sequence prediction, finally, a recognition result is compensated, and network state early warning is generated. According to the invention, the technical problems of insufficient network anomaly detection precision and early warning lag caused by incapability of effectively integrating multi-modal heterogeneous data and adapting to a dynamically changing network environment in real time in the prior art are solved, and multi-modal data fusion and multi-level collaborative analysis driven by big data are achieved. The technical effects of accurate network anomaly detection and prospective early warning are achieved.

Description

Technical Field

[0001] The present invention relates to the field of digital information monitoring technology, and in particular to a big data driven network operation status monitoring management method and system. Background Art

[0002] As the scale and complexity of networks increase, traditional network monitoring systems face multiple challenges. Existing technologies mainly rely on static rules and simple data analysis, which cannot effectively integrate heterogeneous data from different devices, links, and user behaviors, resulting in insufficient accuracy in anomaly detection. In addition, existing systems are difficult to adapt to dynamically changing network environments and lack the ability to predict potential risks. Summary of the invention

[0003] The present application provides a big data-driven network operation status monitoring and management method and system, which is used to solve the technical problem that the existing technology cannot effectively integrate multi-modal heterogeneous data and adapt to the dynamically changing network environment in real time, resulting in insufficient network anomaly detection accuracy and delayed warning.

[0004] The first aspect of the present application provides a big data-driven network operation status monitoring and management method, the method comprising: establishing a distributed database, the distributed database storing multimodal data obtained from network equipment, link status, user behavior and security logs; after time synchronization of the distributed database, establishing an abnormal state multidimensional association graph for the distributed database based on semantic association; establishing a multi-level collaborative network, initializing the multi-level collaborative network based on the abnormal state multidimensional association graph, inputting the time-synchronized distributed database into the multi-level collaborative network, and outputting a collaborative identification result, the multi-level collaborative network comprising a device-level identification layer, a link-level identification layer, a system-level identification layer and a collaborative analysis layer; after locating the anomaly of the distributed database according to the collaborative identification result, executing a time series development forecast and establishing additional abnormal results; after compensating the collaborative identification result according to the additional abnormal result, generating a network status warning.

[0005] The second aspect of the present application provides a big data-driven network operation status monitoring and management system, the system comprising: a database establishment module, the database establishment module is used to establish a distributed database, the distributed database stores multimodal data obtained from network equipment, link status, user behavior and security logs; an abnormal state association module, the abnormal state association module is used to synchronize the distributed database in time, and then establish an abnormal state multidimensional association graph for the distributed database based on semantic association; a multi-level collaborative identification module, the multi-level collaborative identification module is used to establish a multi-level collaborative network, initialize the multi-level collaborative network based on the abnormal state multi-dimensional association graph, input the time-synchronized distributed database into the multi-level collaborative network, and output a collaborative identification result, the multi-level collaborative network includes a device-level identification layer, a link-level identification layer, a system-level identification layer and a collaborative analysis layer; an abnormal development prediction module, the abnormal development prediction module is used to perform time series development prediction and establish additional abnormal results after locating the abnormality of the distributed database according to the collaborative identification results; an identification result compensation module, the identification result compensation module is used to generate a network status warning after compensating the collaborative identification result according to the additional abnormal result.

[0006] One or more technical solutions provided in this application have at least the following technical effects or advantages: The big data-driven network operation status monitoring management method and system provided in the present application relate to the field of digital information monitoring technology. By establishing a distributed database to store multimodal data, time synchronization and semantic association are performed, a multidimensional association diagram of abnormal states is constructed, and collaborative identification is performed using a multi-level collaborative network. Additional abnormal results are generated by combining abnormal location and time series prediction, and finally the identification results are compensated and a network status warning is generated. The technical problem that the existing technology cannot effectively integrate multimodal heterogeneous data and adapt to the dynamically changing network environment in real time, resulting in insufficient network anomaly detection accuracy and delayed warning is solved. The technical effect of accurate network anomaly detection and forward-looking warning is achieved through multimodal data fusion and multi-level collaborative analysis driven by big data. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0008] Figure 1 A flowchart of a big data-driven network operation status monitoring and management method provided in an embodiment of the present application; Figure 2A schematic diagram of the structure of a big data-driven network operation status monitoring and management system provided in an embodiment of the present application.

[0009] Explanation of the accompanying drawings: database establishment module 11, abnormal state association module 12, multi-level collaborative identification module 13, abnormal development prediction module 14, identification result compensation module 15. DETAILED DESCRIPTION

[0010] The present application provides a big data-driven network operation status monitoring and management method and system, which is used to solve the technical problem that the existing technology cannot effectively integrate multi-modal heterogeneous data and adapt to the dynamically changing network environment in real time, resulting in insufficient network anomaly detection accuracy and delayed warning.

[0011] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0012] It should be noted that the terms "first", "second", etc. in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products, or devices.

[0013] Embodiment 1, as Figure 1 As shown, the present application provides a big data driven network operation status monitoring and management method, the method comprising: P10: Establish a distributed database, which stores multimodal data obtained from network devices, link status, user behavior and security logs.

[0014] Specifically, establishing a distributed database is the basis for monitoring the network operation status. By deploying a distributed database system, multimodal data from network devices (such as performance indicators of routers and switches), link status (such as latency, bandwidth, packet loss rate and other parameters reflecting network quality), user behavior (such as access frequency, page click path) and security logs (such as the number of failed logins, malicious IP access records) are uniformly stored and managed. A distributed database is a system that distributes data on multiple nodes and can provide high availability and high scalability. In this step, select a distributed database system that supports multimodal data storage to meet the storage requirements of structured data (such as link indicator tables), semi-structured data (such as user behavior logs in JSON format) and unstructured data (such as security log text).

[0015] First, the data from multiple sources is collected in real time through the data collection module. For example, the CPU utilization and memory occupancy of network devices are obtained through the SNMP (Simple Network Management Protocol) interface, the link status can be generated through traffic monitoring tools (such as NetFlow) to generate data records, user behavior data is collected through website analysis tools, and security logs come from firewalls or IDS (Intrusion Detection System). In order to ensure that these multi-source data can be stored uniformly, the database needs to support timestamp indexing and flexible insertion of multiple data formats, and at the same time distribute the storage to multiple nodes through the sharding mechanism to enhance the system's throughput.

[0016] In addition, to ensure the consistency and time sequence of data between different nodes, the data in the distributed database is uniformly timestamped (such as through the NTP network time protocol) and stored in chronological order, making subsequent time series analysis more accurate. In this way, multimodal data from different sources are integrated into a unified distributed storage platform to form an efficient and flexible data storage system. This design not only improves the real-time performance of data reading, but also provides a high-quality input data source for subsequent anomaly analysis and model prediction.

[0017] P20: After time synchronization of the distributed database, a multi-dimensional association diagram of abnormal status is established for the distributed database based on semantic association.

[0018] Furthermore, step P20 of the embodiment of the present application also includes: P21: Execute attribute feature extraction of the data source and establish an attribute feature set; P22: After standardizing the attribute feature set, calculate the covariance matrix, which represents the linear relationship of the attributes; P23: Perform eigenvalue decomposition on the covariance matrix to obtain eigenvalues ​​and eigenvectors; P24: Use the eigenvalues ​​and eigenvectors to perform principal component selection and establish a key attribute set; P25: Perform semantic analysis on the key attribute set to establish strongly associated attributes, potentially associated attributes and background attributes; P26: Perform mutual information analysis on the key attribute set to establish attribute correlation; P27: Use the strongly associated attributes, the potentially associated attributes, the background attributes and attribute correlation to establish a multidimensional association diagram for abnormal states.

[0019] It should be understood that after completing the data time synchronization of the distributed database, building a multi-dimensional association graph of abnormal states based on semantic associations requires a series of steps to gradually extract data features, analyze attribute relationships, and finally build an association graph. This process starts with attribute feature extraction, gradually builds a logical relationship network of data, and finally presents the abnormal associations between data intuitively.

[0020] First, perform attribute feature extraction of the data source, that is, extract the core attributes of each type of data from the distributed database. For example, extract CPU utilization and memory occupancy from network device data, extract packet loss rate and latency from link status, extract access frequency and path depth from user behavior, and extract the number of abnormal login events from security logs. The extracted attribute features are stored as an attribute feature set, which is a set of key attributes of all source data.

[0021] Next, the attribute feature set is standardized, that is, the attributes of different dimensions and ranges are unified to comparable scales, such as normalizing CPU utilization (0100%) and packet loss rate (01) to the same range according to the maximum and minimum values. The standardized data is used to calculate the covariance matrix, which is a mathematical tool for measuring the linear relationship between different attributes. For example, if the increase in CPU utilization of a device is positively correlated with the increase in its network latency, their values ​​in the covariance matrix are positive.

[0022] Then, the covariance matrix is ​​decomposed into eigenvalues, which is to simplify the complex attribute relationship into key feature representations from a mathematical perspective. The eigenvalue represents the importance of each principal component, and the eigenvector represents the contribution of each attribute in the principal component. For example, if the first principal component is associated with the CPU utilization and link delay of the device, the eigenvector can be used to quantify its weight.

[0023] Furthermore, principal component selection is performed based on eigenvalues ​​and eigenvectors to select the most critical attributes for abnormal status from a large number of attributes to form a key attribute set. For example, through analysis, it may be found that CPU utilization, packet loss rate, and abnormal login events are the most representative attributes for the current system abnormality.

[0024] Next, semantic analysis is performed on the key attribute set. By analyzing the causal relationship and business background between attributes, the attributes are divided into strongly associated attributes (such as CPU utilization is closely related to latency), potentially associated attributes (such as user access frequency may affect device load), and background attributes (such as the potential impact of weather conditions on network stability).

[0025] Subsequently, combined with mutual information analysis, the degree of information sharing between key attributes is further quantified, that is, the nonlinear correlation between attributes is analyzed. For example, through mutual information analysis, it may be found that there is a hidden nonlinear correlation between the increase in packet loss rate and the access behavior of a specific user.

[0026] Finally, a multidimensional association graph of abnormal states is established using strong correlation attributes, potential correlation attributes, background attributes, and the attribute correlation obtained by analysis. This is a model represented by a graph structure, in which nodes represent key attributes and edges represent the strength of the association between them. For example, one node represents CPU utilization, and another node represents link packet loss rate, and the edge weight between the two represents their association strength. This multidimensional association graph can not only present the abnormal propagation path of the current system, but also reflect the potential implicit associations, providing deeper insights for abnormal analysis.

[0027] Through the above steps, the constructed multi-dimensional correlation diagram of abnormal states realizes the intuitive visualization of complex multimodal data relationships, reveals the propagation mechanism of abnormal states, and provides scientific support for subsequent abnormality positioning and risk prediction.

[0028] Furthermore, step P27 of the embodiment of the present application also includes: P27-1: Use the Kafka stream processing framework to capture the incoming data in the distributed database in real time; P27-2: Perform association calculations on the nodes and edges in the abnormal state multidimensional relationship graph based on the incoming data, and dynamically update the graph structure according to the association calculation results.

[0029] Optionally, in order to achieve real-time dynamic update of the multi-dimensional association graph of abnormal status, the present application further introduces the Kafka stream processing framework and dynamic graph structure update mechanism to cope with the continuous influx of new data in the distributed database, ensuring that the association graph always reflects the latest network operation status and potential anomalies.

[0030] First, the Kafka stream processing framework is used to capture the incoming data in the distributed database in real time. Kafka is a high-throughput distributed message queue system designed specifically for processing real-time data streams. It continuously receives data inflows from distributed databases through the model of producers and consumers. For example, when a network device uploads new performance indicators (such as increased CPU utilization) or a user behavior triggers a security event log (such as multiple login failures), Kafka will immediately write this data as a message to its topic and forward it to the graph update module for analysis through the stream processing pipeline.

[0031] Next, the nodes and edges in the association graph are associated with each other based on the incoming data. In a multidimensional association graph, nodes represent key attributes (such as CPU utilization, packet loss rate, etc.), and edges represent the strength of association between attributes. The core of the association calculation is to use an incremental update algorithm to adjust only the nodes and edges affected by new data without rebuilding the entire graph structure. For example, when new security log data indicates that an IP address is associated with multiple abnormal login events, the system will update the "IP address" node and its edge weight with the "abnormal login number" node. If the new data also reveals the potential impact of a user's access behavior on device performance indicators, new nodes and edges will be dynamically added to incorporate these relationships into the graph in real time.

[0032] To achieve this dynamic update, a streaming computing strategy based on sliding windows can be used. The sliding window ensures that the calculation is only for recent data flows by defining a time range (for example, the last 5 minutes), thereby improving processing efficiency and ensuring the real-time nature of the graph structure. In addition, to improve calculation accuracy, the system uses weighted mutual information analysis to quantify the impact of new data on the association of existing nodes and edges. For example, new packet loss rate data may cause the edge weight between a link delay node and a packet loss rate node to increase, reflecting a closer association.

[0033] Finally, the graph structure is dynamically updated based on the correlation calculation results to ensure that the multi-dimensional correlation graph can reflect the abnormal propagation path in network operation in real time. For example, if a link experiences a sudden delay and causes multiple users to experience a decline, the system will dynamically add related nodes and edges to mark the key points and impact range of the anomaly in real time. This update mechanism enables the multi-dimensional correlation graph to quickly adapt to environmental changes and provide more accurate anomaly location and risk warning capabilities.

[0034] By introducing the Kafka stream processing framework and dynamic update algorithm, this step significantly improves the system's ability to process real-time data, making the multi-dimensional association graph efficient and flexible in its adaptive characteristics. In this way, the monitoring of the network operation status is not only global, but also able to capture local emergencies, providing technical support for timely response measures.

[0035] P30: Establish a multi-level collaborative network. After initializing the multi-level collaborative network based on the abnormal state multi-dimensional association diagram, input the time-synchronized distributed database into the multi-level collaborative network and output the collaborative identification result. The multi-level collaborative network includes a device-level identification layer, a link-level identification layer, a system-level identification layer and a collaborative analysis layer.

[0036] Furthermore, the distributed database after time synchronization is input into the multi-level collaborative network, and the collaborative recognition result is output. Step P30 of the embodiment of the present application also includes: P31: Use the device-level identification layer to perform device anomaly analysis on the distributed database as follows: ;in, Characterization equipment In time The abnormal status score is used to quantify the health status or abnormality of the device. is the total number of device features, which indicates the number of features used to describe the device status, such as temperature, pressure, current, etc. Characterize any device feature, Indicates the importance of each feature in abnormal state analysis. The weight is determined by historical data or expert knowledge. Characterization The characteristic value of each device characteristic, Characterizing Device Characteristics In the time window The trend function within the time window is used to describe the characteristic value Changes within is the weight of the historical anomaly pattern matching factor, which determines the impact of historical data on the current anomaly score. Characterization equipment The matching score of the historical abnormal pattern library can measure the similarity between the current state and the historical abnormal pattern. is the weight of the equipment synergy impact factor, is the global synergistic impact factor.

[0037] It should be understood that the establishment of a multi-level collaborative network is the core link for analyzing the abnormal state of the system. It is based on the multi-dimensional association diagram of the abnormal state. By initializing the multi-level collaborative network and inputting the time-synchronized distributed database, it can realize comprehensive abnormality identification from the device level to the system level. The multi-level collaborative network includes four main levels: device-level identification layer, link-level identification layer, system-level identification layer and collaborative analysis layer. Each level works together to jointly output collaborative identification results.

[0038] For example, the core of equipment abnormality analysis is to use formulas to quantify the abnormal state of equipment, extract multi-dimensional feature data of equipment from the distributed database, and calculate the characteristics of each feature in the time window in turn. The changing trend within For example, if the voltage of a device fluctuates abnormally in the past 10 minutes, the trend function will capture this dynamic characteristic. Then, the current state of the device is matched with the historical abnormal pattern library, and the matching score is calculated through the pattern matching algorithm. For example, if the vibration frequency of a device matches the previously recorded mechanical failure mode, the matching score is high. Next, the multi-dimensional correlation diagram of abnormal states is used to analyze the synergistic relationship between devices and calculate the global synergistic impact factor of the device. , evaluate the potential impact of the device anomaly on other devices or links. Finally, combine all the calculation results, calculate the device anomaly status score through feature weight, matching factor weight, and synergy factor weight, and generate device-level anomaly analysis results.

[0039] Through the analysis of the device-level recognition layer, the potential abnormal state of a single device can be quickly identified, and the coverage and depth of anomaly detection can be expanded by combining collaborative relationships. This analysis result provides accurate input for the link-level recognition layer, ensuring that the anomaly detection logic of the multi-level collaborative network is gradually optimized from low to high.

[0040] The device-level identification results will be synchronized to the collaborative analysis layer, where context analysis and anomaly propagation mechanisms are used to comprehensively analyze the device-level, link-level, and system-level results, and ultimately output global collaborative identification results. This process not only improves the accuracy of anomaly detection, but also reduces the impact of single-point failures on the system through multi-layer collaborative mechanisms, ensuring efficient operation of the entire network and real-time anomaly response capabilities.

[0041] Furthermore, step P32 of the embodiment of the present application also includes: P32-1: Use the link level identification layer to identify link anomalies in the distributed database as follows: ;in, Characterization Link In time The interaction anomaly score of Characterization and Links The set of adjacent links, Characterize any link Adjacent links, Characterization Link With link The interaction weights, Characterization Link With link The interaction function of Characterizes the abnormal score of the link itself. For Link The anomaly propagation factor characterizes the diffusion effect of anomalies on adjacent links. and They are the weights of their own characteristic anomalies and anomaly propagation respectively; P32-2: Establish link anomaly identification results, perform system anomaly identification based on a distributed database through the system-level identification layer, establish system anomaly identification results, synchronize the link anomaly identification results, the system anomaly identification results, and the device anomaly identification results to the collaborative analysis layer to establish collaborative identification results.

[0042] In a possible embodiment of the present application, first at the link-level identification layer, the above-mentioned anomaly score calculation formula is used to perform detailed analysis and interactive anomaly scoring of the link status information in the distributed database, calculate the interactive anomaly score of each link, and compare the anomaly score threshold to identify potential link anomalies and establish a link anomaly identification result. The anomaly score threshold can be set based on empirical data.

[0043] Then, after completing the link-level anomaly analysis, the entire network system can be further identified for anomalies through the system-level identification layer. The system-level identification layer considers not only the status of a single link, but also the interaction between links, the status of the device, and the overall health of the system. Afterwards, the link-level anomaly identification results, the device-level anomaly identification results, and the system-level anomaly identification results are synchronized to the collaborative analysis layer. At the collaborative analysis layer, the identification results of each level are comprehensively analyzed to form a comprehensive collaborative identification result, reflecting the anomalies of each element in the network and their mutual influence. Through this hierarchical analysis, this solution can accurately identify network anomalies from multiple levels and multiple angles, and can promptly discover potential chain effects within the system, providing comprehensive network security protection.

[0044] P40: After locating the anomaly of the distributed database according to the collaborative identification results, perform time series development prediction and establish additional anomaly results.

[0045] Furthermore, step P40 of the embodiment of the present application also includes: P41: Create a time zero point based on the collaborative identification result; P42: Perform data backtracking extraction based on the time zero point and anomaly location, perform time series development prediction based on the data backtracking extraction result, and establish the additional anomaly result.

[0046] Optionally, based on the collaborative identification results obtained in the previous step, first locate the anomaly of the distributed database, then perform time series development prediction, and finally establish additional anomaly results. The core of this step is to accurately locate anomalies and combine time series analysis of historical data to predict future anomaly trends, ensuring that the network management system can identify potential risks in advance.

[0047] First, the results of collaborative identification are used to locate anomalies in the network. This location can identify the specific source of the anomaly, which may involve devices, links, or the system itself. This process is critical to reducing false positives because it helps the system focus resources on the most urgent and critical anomaly analysis rather than wasting energy on irrelevant anomalies.

[0048] Next, based on the positioning results, the time series development prediction phase is entered. Time series development prediction is to predict possible abnormal changes in the future by analyzing the trend of historical data. Here, the system will use data backtracking, that is, starting from a certain time zero, backtracking to previous data, and extracting relevant time series data. These data provide the basis for subsequent forecasting analysis, thereby ensuring the accuracy and timeliness of the forecast. Through forecasting models (such as ARIMA or LSTM), not only the current abnormal changes are paid attention to, but also their possible expansion and duration are deeply analyzed, providing managers with a more comprehensive network status warning.

[0049] After executing the time series forecast, additional abnormal results will be generated, which can reveal potential risk points and change trends in the future. The introduction of additional abnormal results enables the system to identify possible problems before the abnormality occurs, providing relevant personnel with enough time to take precautions. This not only enhances the system's early warning capabilities, but also optimizes resource allocation and risk management strategies.

[0050] In general, by combining anomaly location and time series prediction, a forward-looking and highly accurate early warning mechanism is provided for the network management system. It can identify and predict abnormal events that may affect network stability in a short period of time, thereby providing decision support for operations and maintenance personnel, taking necessary intervention measures in advance, and ensuring the stable operation of the network.

[0051] P50: After compensating the collaborative identification result according to the additional abnormal result, a network status warning is generated.

[0052] Specifically, according to the generated additional abnormal results, the previously obtained collaborative identification results are compensated, and finally a network status warning is generated. The purpose of this process is to make the final warning more accurate and timely by correcting the predicted abnormal trend, thereby providing more reliable decision support for network operation and maintenance.

[0053] First, the collaborative identification results are compensated with additional anomaly results. The additional anomaly results are derived from time series predictions and provide forecast information about the future state of the network, which can help the system capture potential anomalies that may be missed by the initial collaborative identification results. In this step, the system combines the additional anomaly results with the previous collaborative identification results to correct and adjust the existing identification results to ensure that the collaborative identification results more accurately reflect the current health of the network after taking into account future development trends.

[0054] By compensating the recognition results, errors caused by prediction lag or data fluctuations can be eliminated, thereby reducing the probability of false alarms and improving the accuracy of anomaly detection. For example, in some cases, the initial collaborative recognition may fail to capture the potential abnormal change trend of certain devices or links, while the additional abnormal results can timely identify and correct these omissions, thereby avoiding missing important early warning signals.

[0055] Finally, the compensated recognition results are used to generate network status warnings. This warning is based on more accurate and complete abnormal analysis results, which can accurately reflect the current network operation status and its potential risks. Network status warnings not only provide immediate abnormal warnings, but also prioritize according to the severity of the warnings, providing managers with reasonable response strategies. For example, based on the compensated results, warnings about equipment failures, link congestion, or system bottlenecks can be issued to help operation and maintenance personnel intervene in time before the problem expands, thereby improving the accuracy and timeliness of warnings.

[0056] Furthermore, the embodiment of the present application further includes step P60, and step P60 further includes: P61: Establish a network status monitoring database; P62: Perform early warning superposition analysis based on the network status monitoring database and the network status early warning, and issue a warning according to the early warning superposition analysis result.

[0057] In a possible embodiment of the present application, a network status monitoring database can be further established, and a warning superposition analysis can be performed based on the database and the network status warning, and finally a warning alarm can be generated according to the analysis results. That is, through multi-dimensional information integration and analysis, a more comprehensive and detailed warning report can be provided for network management, thereby improving the response capability and accuracy to potential problems.

[0058] First, a network status monitoring database is built to store network operation information from different data sources. The database collects real-time operation data, link status, user behavior, historical warning records, etc. of various network devices. These data provide strong support for subsequent analysis. The core function of the monitoring database is to continuously update and save the dynamic status information of the network, providing a unified data platform for the entire warning system.

[0059] Next, based on the various data stored in the monitoring database and the previously generated network status warnings, the warning overlay analysis is performed. This analysis aims to integrate warning information from different levels and sources, and identify more complex or potential risks through multi-dimensional cross-validation. For example, an abnormal warning of a device may be related to factors such as excessive link load or insufficient system processing capacity. The warning overlay analysis will synthesize this information to dig out potential chain reactions or more complex system failure modes.

[0060] After the early warning overlay analysis is completed, a warning alert will be generated based on the analysis results. These warning alerts will provide detailed warning information based on different abnormal levels and impact ranges, and provide corresponding response suggestions for relevant personnel. These alert information can help network managers understand the current health status of the network, potential problems and possible solutions, so as to intervene in time and prevent the problem from further deteriorating.

[0061] In summary, the embodiments of the present application have at least the following technical effects: This application establishes a distributed database to store multimodal data from network devices, link status, user behavior and security logs, and performs time synchronization and semantic association to build a multidimensional association graph of abnormal states. Subsequently, a multi-level collaborative network (including device-level, link-level, system-level recognition layer and collaborative analysis layer) is used for collaborative recognition, and additional abnormal results are generated by combining abnormal location and time series prediction. Finally, the recognition results are compensated, and a network status warning is generated to achieve accurate detection and prediction of network anomalies.

[0062] The technical effect of achieving accurate network anomaly detection and forward-looking early warning has been achieved through multimodal data fusion and multi-level collaborative analysis driven by big data.

[0063] Embodiment 2, based on the same inventive concept as the network operation status monitoring and management method driven by big data in the above embodiment, Figure 2 As shown, the present application provides a big data driven network operation status monitoring and management system, and the system and method embodiments in the present application embodiments are based on the same inventive concept. The system includes: The database establishment module 11 is used to establish a distributed database, and the distributed database stores multimodal data obtained from network devices, link status, user behavior and security logs.

[0064] The abnormal state association module 12 is used to establish an abnormal state multi-dimensional association graph for the distributed database based on semantic association after time synchronization of the distributed database.

[0065] A multi-level collaborative identification module 13 is used to establish a multi-level collaborative network. After the multi-level collaborative network is initialized based on the abnormal state multi-dimensional association diagram, the time-synchronized distributed database is input into the multi-level collaborative network, and the collaborative identification result is output. The multi-level collaborative network includes a device-level identification layer, a link-level identification layer, a system-level identification layer and a collaborative analysis layer.

[0066] The abnormal development prediction module 14 is used to perform time series development prediction and establish additional abnormal results after locating the abnormality of the distributed database according to the collaborative identification results.

[0067] The recognition result compensation module 15 is used to generate a network status warning after compensating the collaborative recognition result according to the additional abnormal result.

[0068] Furthermore, the abnormal state association module 12 is further configured to perform the following steps: Execute attribute feature extraction of the data source and establish an attribute feature set; after standardizing the attribute feature set, calculate the covariance matrix, the covariance matrix represents the linear relationship of the attributes; perform eigenvalue decomposition on the covariance matrix to obtain eigenvalues ​​and eigenvectors; use the eigenvalues ​​and eigenvectors to perform principal component selection and establish a key attribute set; perform semantic analysis on the key attribute set to establish strongly associated attributes, potentially associated attributes and background attributes; perform mutual information analysis on the key attribute set to establish attribute correlation; use the strongly associated attributes, the potentially associated attributes, the background attributes and attribute correlation to establish a multidimensional association diagram of abnormal states.

[0069] Furthermore, the abnormal state association module 12 is further configured to perform the following steps: Use the Kafka stream processing framework to capture incoming data in distributed databases in real time; Based on the inflow data, association calculations are performed on nodes and edges in the abnormal state multidimensional relationship graph, and the graph structure is dynamically updated according to the association calculation results.

[0070] Furthermore, the multi-level collaborative identification module 13 is further configured to perform the following steps: The device-level identification layer is used to perform device anomaly analysis of the distributed database as follows: ;in, Characterization equipment In time Abnormal status score, is the total number of device features, Characterize any device feature, Characterizing Device Characteristics The weight of Characterization The characteristic value of each device characteristic, Characterizing Device Characteristics In the time window The changing trend function within is the weight of the historical abnormal pattern matching factor, Characterization equipment The matching score of the historical abnormal pattern library, is the weight of the equipment synergy impact factor, is a global collaborative influencing factor; establishes a device anomaly identification result, synchronizes the device anomaly identification result to the collaborative analysis layer, and establishes a collaborative identification result.

[0071] Furthermore, the multi-level collaborative identification module 13 is further configured to perform the following steps: The link level identification layer is used to identify link anomalies in a distributed database as follows: ;in, Characterization Link In time The interaction anomaly score of Characterization and Links The set of adjacent links, Characterize any link Adjacent links, Characterization Link With link The interaction weights, Characterization Link With link The interaction function of Characterizes the abnormal score of the link itself. For Link The anomaly propagation factor characterizes the diffusion effect of anomalies on adjacent links. and They are the weights of their own characteristic anomalies and anomaly propagation respectively; establish a link anomaly identification result, perform system anomaly identification based on a distributed database through a system-level identification layer, establish a system anomaly identification result, synchronize the link anomaly identification result, the system anomaly identification result, and the device anomaly identification result to the collaborative analysis layer to establish a collaborative identification result.

[0072] Furthermore, the abnormal development prediction module 14 is also used to perform the following steps: A time zero point is created based on the collaborative identification result; data backtracking extraction is performed based on the time zero point and the anomaly location, and a time series development forecast is performed according to the data backtracking extraction result to establish the additional anomaly result.

[0073] Furthermore, the system also includes: A network status monitoring database establishment module, which is used to establish a network status monitoring database; an early warning superposition analysis module, which is used to perform early warning superposition analysis based on the network status monitoring database and the network status early warning, and issue early warnings according to the early warning superposition analysis results.

[0074] It should be noted that the above-mentioned sequence of the embodiments of the present application is only for description and does not represent the advantages and disadvantages of the embodiments. And the above-mentioned specific embodiments of this specification are described. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0075] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

[0076] This specification and the drawings are merely exemplary illustrations of the present application and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, a person skilled in the art may make various modifications and variations to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present application and its equivalents, the present application intends to include these modifications and variations.

Claims

1. A network operation status monitoring and management method driven by big data, characterized in that: The method comprises: Establishing a distributed database storing multimodal data obtained from network devices, link status, user behavior, and security logs; After time synchronization of the distributed database, a multi-dimensional association graph of abnormal status is established for the distributed database based on semantic association; Establish a multi-level collaborative network, initialize the multi-level collaborative network based on the abnormal state multi-dimensional association graph, input the time-synchronized distributed database into the multi-level collaborative network, and output the collaborative identification result, wherein the multi-level collaborative network includes a device-level identification layer, a link-level identification layer, a system-level identification layer and a collaborative analysis layer; After locating the anomaly of the distributed database according to the collaborative identification results, a time series development forecast is performed to establish additional anomaly results; After compensating the collaborative identification result according to the additional abnormal result, a network status warning is generated.

2. The big data driven network operation status monitoring and management method according to claim 1, characterized in that: The method of establishing a multi-dimensional association graph of abnormal states for a distributed database based on semantic associations includes: Execute attribute feature extraction of data source and establish attribute feature set; After the attribute feature set is standardized, a covariance matrix is ​​calculated, wherein the covariance matrix represents the linear relationship of the attributes; Performing eigenvalue decomposition on the covariance matrix to obtain eigenvalues ​​and eigenvectors; Performing principal component selection using the eigenvalues ​​and the eigenvectors to establish a key attribute set; Performing semantic analysis on the key attribute set to establish strong correlation attributes, potential correlation attributes and background attributes; Performing mutual information analysis on the key attribute set to establish attribute correlation; A multi-dimensional association diagram of abnormal states is established by using the strong association attributes, the potential association attributes, the background attributes and the attribute correlation.

3. The big data driven network operation status monitoring and management method according to claim 2, characterized in that: The establishing of the abnormal state multi-dimensional association diagram also includes: Use the Kafka stream processing framework to capture incoming data in distributed databases in real time; Based on the inflow data, association calculations are performed on nodes and edges in the abnormal state multidimensional relationship graph, and the graph structure is dynamically updated according to the association calculation results.

4. The big data driven network operation status monitoring and management method according to claim 1, characterized in that: The step of inputting the time-synchronized distributed database into the multi-level collaborative network and outputting collaborative recognition results includes: The device-level identification layer is used to perform device anomaly analysis of the distributed database as follows: ； in, Characterization equipment In time Abnormal status score, is the total number of device features, Characterize any device feature, Characterizing Device Characteristics The weight of Characterization The characteristic value of each device characteristic, Characterizing Device Characteristics In the time window The changing trend function within is the weight of the historical abnormal pattern matching factor, Characterization equipment The matching score of the historical abnormal pattern library, is the weight of the equipment synergy impact factor, is the global synergistic impact factor; Establish a device anomaly identification result, synchronize the device anomaly identification result to the collaborative analysis layer, and establish a collaborative identification result.

5. The big data driven network operation status monitoring and management method according to claim 4, characterized in that: The step of synchronizing the device abnormality identification result to the collaborative analysis layer to establish the collaborative identification result includes: The link level identification layer is used to identify link anomalies in a distributed database as follows: ； in, Characterization Link In time The interaction anomaly score of Characterization and Links The set of adjacent links, Characterize any link Adjacent links, Characterization Link With link The interaction weights, Characterization Link With link The interaction function of Characterizes the abnormal score of the link itself. For Link The anomaly propagation factor characterizes the diffusion effect of anomalies on adjacent links. and are the weights of their own characteristic anomaly and anomaly propagation respectively; Establish link anomaly identification results, perform system anomaly identification based on a distributed database through a system-level identification layer, establish system anomaly identification results, synchronize the link anomaly identification results, the system anomaly identification results, and the device anomaly identification results to the collaborative analysis layer, and establish collaborative identification results.

6. The big data driven network operation status monitoring and management method according to claim 1, characterized in that: After locating the anomaly of the distributed database according to the collaborative identification result, executing the time series development prediction and establishing the additional anomaly result include: Creating a time zero point based on the collaborative recognition result; Data back-extraction is performed based on the time zero point and the anomaly location, and time series development prediction is performed according to the data back-extraction result to establish the additional anomaly result.

7. The big data driven network operation status monitoring and management method according to claim 1, characterized in that: The method further comprises: Establish a network status monitoring database; Based on the network status monitoring database and the network status warning, a warning superposition analysis is performed, and a warning alarm is issued according to the warning superposition analysis result.

8. A big data driven network operation status monitoring and management system, characterized in that: The system comprises: A database establishment module, the database establishment module is used to establish a distributed database, the distributed database stores multimodal data obtained from network devices, link status, user behavior and security logs; An abnormal state association module, wherein the abnormal state association module is used to establish an abnormal state multi-dimensional association graph for the distributed database based on semantic association after time synchronization of the distributed database; A multi-level collaborative identification module, the multi-level collaborative identification module is used to establish a multi-level collaborative network, after the multi-level collaborative network is initialized based on the abnormal state multi-dimensional association graph, the time-synchronized distributed database is input into the multi-level collaborative network, and the collaborative identification result is output, the multi-level collaborative network includes a device-level identification layer, a link-level identification layer, a system-level identification layer and a collaborative analysis layer; An abnormal development prediction module, which is used to perform time series development prediction and establish additional abnormal results after locating the abnormality of the distributed database according to the collaborative identification results; The recognition result compensation module is used to generate a network status warning after compensating the collaborative recognition result according to the additional abnormal result.

Images