Flow data monitoring method and device, electronic equipment and program product

Abstract

The invention discloses a flow data monitoring method and device, electronic equipment and a program product, and relates to the field of network security, the monitoring method comprises the following steps: receiving metadata transmitted by different flow equipment, carrying out load balancing processing on all the received metadata to obtain first preset metadata corresponding to each metadata, and sending the first preset metadata to a server; and based on a transmission protocol corresponding to each piece of first preset metadata, sending each piece of first preset metadata to different partitions in a preset message queue, processing the first preset metadata of each partition to obtain second preset metadata, and storing all the second preset metadata in a preset real-time database. And monitoring all the second preset metadata in the preset real-time database, and visually displaying a monitoring result. According to the invention, the technical problem that the efficiency of monitoring a large amount of traffic data is low in the prior art is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of network security, in particular to a traffic data monitoring method and device, electronic equipment and program product. BACKGROUND

[0002] In the field of network security monitoring, real-time processing of massive metadata (such as network traffic logs, device alarms, user behavior trajectories, etc.) and efficient storage and retrieval are the core technical basis for building a proactive defense system. With the development of 5G, Internet of Things and other technologies, the network attack surface is expanding exponentially. The current data processing architecture has the following problems in data storage and monitoring:

[0003] (1) The contradiction between data size and timeliness, a single enterprise network boundary generates a large amount of (such as 10TB+) metadata per day, and needs to complete the whole process of analysis, threat matching and response decision-making within a short time (such as 1 second). The current data processing architecture cannot meet the real-time analysis requirements, and the current stream computing system has design defects in state management and fault recovery, resulting in a high false negative rate for complex attack behavior detection;

[0004] (2) Imbalance between storage and computing resources, raw security metadata contains a large amount of low-value information (such as normal traffic packet features), and direct full storage leads to a sharp increase in storage costs. The current storage system lacks dynamic aggregation capability, has a low compression rate for time series data, and cannot support millisecond-level multi-dimensional aggregation queries (such as "statistical behavior of a certain IP (address) in the past 5 minutes across protocols");

[0005] (3) System scalability and security bottlenecks, single-machine architecture has an upper limit on processing capacity, which cannot cope with data flood during DDoS (Distributed Denial of Service) attacks;

[0006] (4) Uncontrolled operation complexity, the current separated monitoring system requires switching between multiple independent interfaces, and the average time for fault location is long, lacks automatic scaling mechanism, and manual adjustment of resource allocation leads to a large amount of idle computing resources.

[0007] In view of the above problems, no effective solution has been proposed so far. SUMMARY

[0008] The embodiments of the present application provide a traffic data monitoring method and device, electronic equipment and program product to at least solve the technical problem of low efficiency in monitoring a large amount of traffic data in the related art.

[0009] According to an aspect of the embodiments of the present application, a method for monitoring traffic data is provided, which is applied to a data monitoring system and includes: receiving metadata transmitted by different traffic devices, wherein the different traffic devices use different transmission protocols, and the metadata is traffic data transmitted by the traffic devices based on the transmission protocols; performing load balancing processing on all the received metadata to obtain first preset metadata corresponding to each piece of metadata, and sending each piece of first preset metadata to different partitions in a preset message queue based on a transmission protocol corresponding to each piece of first preset metadata; processing the first preset metadata in each partition to obtain second preset metadata, and storing all the second preset metadata in a preset real-time database, wherein the second preset metadata is metadata that meets the field requirements of the corresponding transmission protocol; monitoring all the second preset metadata in the preset real-time database, and visually displaying the monitoring results.

[0010] Further, the data monitoring system at least includes an agent and multiple servers, and the step of performing load balancing processing on all the received metadata to obtain first preset metadata corresponding to each piece of metadata includes: in the case that the agent receives multiple pieces of metadata, storing all the metadata in the multiple servers based on the current load of each server, wherein the agent obtains the metadata by listening to a preset port, and the preset port is a port opened by the data monitoring system to the traffic devices; and preprocessing the metadata stored in each server to obtain the first preset metadata.

[0011] Further, the step of preprocessing the metadata stored in each server to obtain the first preset metadata includes: parsing the metadata to extract multiple groups of key-value pairs in the metadata; cleaning the multiple groups of key-value pairs to obtain multiple groups of initial key-value pairs; normalizing the initial key-value pairs to obtain target key-value pairs; and integrating all the target key-value pairs to obtain the first preset metadata.

[0012] Further, the step of processing the first preset metadata of each partition to obtain the second preset metadata includes: determining field requirements based on the transmission protocol corresponding to the partition, wherein the field requirements include a preset field set, which includes multiple preset fields; parsing the first preset metadata within the partition to obtain multiple primary keys and their corresponding key values; treating each primary key as a field and matching all fields with the preset field set to obtain target fields, wherein the target fields are preset fields that do not exist in the first preset metadata; matching the field values ​​corresponding to the target fields from the preset database; normalizing the key values ​​corresponding to all fields and the field values ​​corresponding to all target fields to obtain target key values ​​corresponding to all fields and target field values ​​corresponding to all target fields; and determining the second preset metadata based on all fields, the target key values ​​corresponding to all fields, all target fields, and the target field values ​​corresponding to all target fields.

[0013] Furthermore, after obtaining the second preset metadata, the method further includes: storing all the second preset metadata in a preset historical database, wherein the preset historical database is used to permanently store the second preset metadata, and the steps of storing the second preset metadata include: establishing a data table corresponding to each second preset metadata in the preset historical database; determining a set of sub-table fields based on the transmission protocol corresponding to each second preset metadata; filtering sub-table data corresponding to each sub-table field from the second preset metadata based on each sub-table field in the sub-table field set; and storing each sub-table data in a sub-table of the data table.

[0014] Furthermore, each second preset metadata item carries a timestamp. The step of storing all second preset metadata items in a preset real-time database includes: dividing the second preset metadata into multiple shards and creating replica data for each shard; establishing a primary key index for each shard based on the timestamp, and establishing a field index for each shard based on the key fields of the second preset metadata, wherein the key fields are fields obtained by analyzing the second preset metadata, and the key fields include at least one of the following: source address field, threat type field; storing each shard associated with the primary key index and field index to multiple storage nodes in the preset real-time database; determining the storage duration of the shard based on the timestamp, and deleting the shard from the preset real-time database when the storage duration of the shard is detected to have reached the valid duration.

[0015] Furthermore, after each first preset metadata is sent to a different partition in the preset message queue, the process also includes: monitoring each partition in the preset message queue; blocking the partition when the monitored data volume stored in the partition reaches the preset data volume; collecting resource data of each partition and determining the resource indicator value of the partition based on the resource data; and blocking the partition when the resource indicator value does not fall within the preset indicator range.

[0016] According to another aspect of the embodiments of the present application, there is also provided a flow data monitoring device applied to a data monitoring system, comprising: a receiving unit configured to receive metadata transmitted by different flow devices, wherein the transmission protocols used by the different flow devices are different, and the metadata is flow data transmitted by the flow devices based on the transmission protocols; a first processing unit configured to perform load balancing processing on all the received metadata to obtain first preset metadata corresponding to each of the metadata, and send each of the first preset metadata to different partitions in a preset message queue based on the transmission protocol corresponding to each of the first preset metadata; a second processing unit configured to process the first preset metadata in each of the partitions to obtain second preset metadata, and store all the second preset metadata to a preset real-time database, wherein the second preset metadata is metadata meeting the field requirements corresponding to the transmission protocol; and a monitoring unit configured to monitor all the second preset metadata in the preset real-time database and visually display the monitoring results.

[0017] Further, the data monitoring system comprises at least a proxy and a plurality of servers, and the first processing unit comprises: a first storage module configured to, in the case that the proxy receives a plurality of metadata, store all the metadata to the plurality of servers based on the current load of each of the servers, wherein the proxy acquires the metadata by listening to a preset port, and the preset port is a port opened by the data monitoring system to the flow devices; and a first processing module configured to pre-process the metadata stored in each of the servers to obtain the first preset metadata.

[0018] Further, the first processing module comprises: a first analysis submodule configured to analyze the metadata to extract a plurality of groups of key-value pairs in the metadata; a first cleaning submodule configured to clean the plurality of groups of key-value pairs to obtain a plurality of groups of initial key-value pairs; a first normalization submodule configured to normalize the initial key-value pairs to obtain target key-value pairs; and a first integration submodule configured to integrate all the target key-value pairs to obtain the first preset metadata.

[0019] Further, the second processing unit comprises: a first determining module, configured to determine a field requirement based on a transmission protocol corresponding to the partition, wherein the field requirement comprises a preset field set, and the preset field set comprises a plurality of preset fields; a first analyzing module, configured to analyze first preset metadata in the partition to obtain a plurality of primary keys and key values corresponding to the primary keys; a first matching module, configured to take each primary key as a field, and match all the fields with the preset field set to obtain target fields, wherein the target fields are preset fields that do not exist in the first preset metadata; a second matching module, configured to match field values corresponding to the target fields from a preset database; a second processing module, configured to perform normalization processing on the key values corresponding to all the fields and the field values corresponding to all the target fields to obtain target key values corresponding to all the fields and target field values corresponding to all the target fields; and a second determining module, configured to determine second preset metadata based on all the fields, the target key values corresponding to all the fields, all the target fields, and the target field values corresponding to all the target fields.

[0020] Further, the monitoring device further comprises: a second storage module, configured to store all the second preset metadata to a preset historical database after obtaining the second preset metadata, wherein the preset historical database is used to permanently store the second preset metadata, and the step of storing the second preset metadata comprises: establishing a data table corresponding to each second preset metadata in the preset historical database; determining a sub-table field set based on a transmission protocol corresponding to each second preset metadata; screening sub-table data corresponding to each sub-table field from the second preset metadata based on each sub-table field in the sub-table field set; and storing each sub-table data in a sub-table in the data table.

[0021] Further, each second preset metadata carries a timestamp, and the second processing unit further comprises: a first dividing module, configured to divide the second preset metadata into a plurality of shards, and create replica data of each shard; a first establishing module, configured to establish a primary key index of each shard based on the timestamp, and establish a field index of each shard based on a key field of the second preset metadata, wherein the key field is a field obtained by analyzing the second preset metadata, and the key field comprises at least one of the following: a source address field, a threat type field; a third storage module, configured to store each shard associated with the primary key index and the field index to a plurality of storage nodes in a preset real-time database; and a third determining module, configured to determine a storage duration of the shard based on the timestamp, and delete the shard from the preset real-time database when it is detected that the storage duration of the shard reaches an effective duration.

[0022] Further, the monitoring device further comprises: a first monitoring module, configured to monitor each partition in the preset message queue after each first preset metadata is sent to the different partition in the preset message queue; a first blocking module, configured to block the partition in the case that the amount of data stored in the partition reaches the preset data amount; a first collection module, configured to collect resource data of each partition, and determine a resource index value of the partition based on the resource data; and a second blocking module, configured to block the partition in the case that the resource index value does not belong to a preset index range.

[0023] According to another aspect of the embodiments of the present application, a computer program product is also provided, comprising a non-volatile computer readable storage medium, the non-volatile computer readable storage medium storing a computer program, the computer program being executed by a processor to implement the monitoring method of the traffic data according to any one of the above.

[0024] According to another aspect of the embodiments of the present application, an electronic device is also provided, comprising one or more processors and a memory, the memory being configured to store one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the monitoring method of the traffic data according to any one of the above.

[0025] In the present application, the metadata transmitted by different traffic devices is received, load balancing processing is performed on all received metadata, first preset metadata corresponding to each metadata is obtained, each first preset metadata is sent to different partitions in a preset message queue based on a transmission protocol corresponding to each first preset metadata, the first preset metadata of each partition is processed to obtain second preset metadata, and all second preset metadata is stored in a preset real-time database. The second preset metadata in the preset real-time database is monitored, and the monitoring result is visually displayed, thereby solving the technical problem of low efficiency in monitoring a large amount of traffic data in the related art.

[0026] In the present application, through the load balancing processing framework in the data monitoring system, the metadata transmitted by different traffic devices can be efficiently processed, and each first preset metadata after the load balancing processing is stored in different partitions in a preset message queue in a big data processing computing cluster in the data monitoring system. The first preset metadata of each partition is processed by the big data processing computing cluster, and the processed second preset metadata is stored in a preset real-time database in a big data storage cluster in the data monitoring system, so as to realize real-time and visual monitoring of the traffic data, and achieve the technical effect of efficiently processing and storing the metadata transmitted by different traffic devices. BRIEF DESCRIPTION OF DRAWINGS

[0027] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application. In the drawings:

[0028] Figure 1 is a flow chart of an optional flow data monitoring method according to an embodiment of the application;

[0029] Figure 2 is a schematic diagram of an optional data monitoring flow using a data monitoring system according to an embodiment of the application;

[0030] Figure 3 is a schematic diagram of an optional flow data monitoring device according to an embodiment of the application;

[0031] Figure 4 is a hardware structure block diagram of an electronic device (or mobile device) for a flow data monitoring method according to an embodiment of the application. DETAILED DESCRIPTION

[0032] In order to make the person skilled in the art better understand the application scheme, the technical scheme in the embodiment of the application will be described clearly and completely below in conjunction with the drawings in the embodiment of the application. Obviously, the described embodiments are only a part of the embodiments of the application, not all. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor should belong to the protection scope of the application.

[0033] It should be noted that the terms "first", "second", and the like in the present application are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0034] It should be noted that the information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) collected and related to the present application are all information and data authorized by the user or authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of related data comply with relevant laws, regulations and standards in the relevant region, necessary security measures are taken, public order and good customs are not violated, and appropriate operation entrances are provided for users to choose authorization or refusal. For example, the system and the related users or institutions are provided with an interface, and before obtaining the relevant information, the interface needs to send a request to the aforementioned user or institution, and after receiving the consent information feedback from the aforementioned user or institution, the relevant information is obtained.

[0035] In the present application, a load balancing processing framework, a big data processing computing cluster and a big data storage cluster are constructed, and a monitoring component based on a distributed message queue and a data analysis component based on a distributed storage cluster are constructed, so as to guarantee that the end-to-end delay of complex security event detection is small. In addition, a columnar storage compression algorithm is developed in the storage optimization layer, which effectively reduces the storage space while maintaining the queryability of the original data. In the system coordination layer, deep protocol-level integration of the computing, storage and monitoring modules is realized, and full-link encryption and self-healing capabilities are realized.

[0036] In the present application, a big data processing computing cluster based on the Hadoop (a software framework for distributed storage and processing of large data sets) ecological core is built, and a big data storage cluster based on a distributed architecture is built, which can receive a large amount of metadata based on the load balancing processing framework to complete the distribution, processing and collection of traffic data, and can perform real-time data processing based on the big data processing computing framework, clean and serialize the data, and write the processed data to the distributed message queue. In addition, the data can be extracted, transformed and loaded based on the big data processing computing framework, and the processed data can be persistently stored in the database. In addition, real-time monitoring of the distributed message queue can be realized through the monitoring component based on the distributed message queue, and real-time abnormal monitoring of the data stored in the big data storage cluster can be realized through the data analysis component based on the distributed storage cluster.

[0037] The present application will be described in detail below in conjunction with various embodiments.

[0038] Embodiment one

[0039] According to an embodiment of the present application, an embodiment of a method for monitoring traffic data is provided. It should be noted that the steps shown in the flowcharts of the drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0040] Figure 1 is a flowchart of an optional method for monitoring traffic data according to an embodiment of the present application, as shown in Figure 1 the method comprises the following steps:

[0041] Step S101, receiving metadata transmitted by different traffic devices, wherein the transmission protocols used by different traffic devices are different, and the metadata is traffic data transmitted by the traffic devices based on the transmission protocols.

[0042] In an embodiment of the present application, the data monitoring system can receive raw traffic data collected by network traffic monitoring devices (e.g., full-flow probes), and these data exist in the form of metadata, which contains key information of network communication, such as source IP address, destination IP address, port number, protocol type, packet length, timestamp, etc. Since different security traffic devices may use different transmission protocols (e.g., Ftp (File Transfer Protocol) protocol, Database protocol, smtp (Simple Mail Transfer Protocol) protocol, udp (User Datagram Protocol) protocol, telnet protocol, Dns (Domain Name System) protocol, http (Hypertext Transfer Protocol) protocol, etc.) to send these metadata, the data monitoring system needs to handle multiple protocols. Specifically, a load balancing processing framework can be constructed in the data monitoring system, which can use Nginx (a kind of reverse proxy server) as a reverse proxy, and can balance the data load to multiple collection nodes in the back end according to the source of the traffic and the protocol type, to ensure that data loss will not occur due to excessive pressure on a single point, and also to ensure the continuity of the session.

[0043] Step S102, performing load balancing processing on all received metadata to obtain a first preset metadata corresponding to each metadata, and sending each first preset metadata to different partitions in a preset message queue based on the transmission protocol corresponding to each first preset metadata.

[0044] In the embodiment of the present application, Nginx can first balance all the received metadata to multiple servers, and then the Spring Boot (a framework) collection service (built into each server of the load balancing processing framework) decodes and structures the metadata to form standardized first preset metadata. This is because the original metadata is usually disorganized, and through standardization processing, data of different devices and protocols can be converted into a unified format for subsequent processing. Subsequently, these first preset metadata are sent to different partitions of a preset message queue (such as a Kafka message queue).

[0045] Here, the partition strategy is usually based on a certain key field (such as the transmission protocol type) of the metadata to ensure that metadata of the same characteristics are aggregated in the same partition, thereby facilitating the locality and parallelism of data in subsequent processing, not only realizing the preliminary cleaning of the metadata, but also preparing for subsequent real-time processing and storage.

[0046] Step S103, processing the first preset metadata of each partition to obtain second preset metadata, and storing all the second preset metadata to a preset real-time database, wherein the second preset metadata is metadata meeting the field requirements of the transmission protocol.

[0047] In the embodiment of the present application, the first preset metadata stored in the Kafka message queue can be processed in real time by Flink (a distributed stream processing framework) in the big data processing computing cluster in the data monitoring system, including further cleaning of data (such as removing invalid or duplicate data), serialization (converting data into storage or transmission format), and necessary conversion (such as converting non-standard timestamps into standard format). After processing, the data is called second preset metadata, which is not only cleaner and more organized, but also meets the field requirements of the specific transmission protocol, that is, the processed metadata has been formatted to be directly written into a distributed storage system (such as ClickHouse (a preset real-time database, that is, an open source columnar database management system for online analytical processing)). By writing data into the real-time database, not only is the persistent storage of data realized, but also the immediate availability of data is ensured, which is crucial for real-time analysis and abnormal alarm.

[0048] Step S104, monitoring all the second preset metadata in the preset real-time database, and visually displaying the monitoring results.

[0049] In the embodiments of the present application, the second preset metadata stored in the ClickHouse real-time database is comprehensively monitored by the monitoring tool, including but not limited to data writing speed, query delay, data integrity and other key performance indicators. These monitoring data are collected and transmitted to a visual dashboard such as Metabase (a data visualization tool) for graphical display, facilitating relevant personnel to quickly identify system status and potential problems. Through real-time monitoring and visualization, abnormalities in data processing can be discovered and addressed in a timely manner, ensuring the healthy operation of the data processing link and also serving as an important guarantee for system security and operation and maintenance.

[0050] In summary, through the load balancing processing framework in the data monitoring system, efficient load balancing processing can be performed on the metadata transmitted by different traffic devices, and each first preset metadata after load balancing processing is stored in different partitions of a preset message queue in a big data processing computing cluster in the data monitoring system. The first preset metadata in each partition is processed by the big data processing computing cluster, and the second preset metadata after processing is stored in a preset real-time database in a big data storage cluster in the data monitoring system, to realize real-time and visual monitoring of traffic data, achieving the purpose of efficient processing and storage of metadata transmitted by different traffic devices, thereby realizing the technical effect of real-time analysis and monitoring of network security metadata, and further solving the technical problem of low efficiency in monitoring a large amount of traffic data in related technologies.

[0051] Optionally, the data monitoring system comprises at least a proxy and multiple servers. To improve the accuracy of load balancing processing of metadata, in the traffic data monitoring method provided in Embodiment One of the present application, when the proxy receives multiple metadata, the metadata are stored in the multiple servers based on the current load of each server. The proxy obtains the metadata by listening to a preset port, and the preset port is a port opened by the data monitoring system to the traffic device. The metadata stored in each server are preprocessed to obtain first preset metadata.

[0052] In the embodiments of the present application, the data monitoring system comprises at least a proxy Nginx and multiple servers. The proxy listens to a preset port by configuration, serving as an entrance for communication between the data monitoring system and the traffic device. The preset port is a port set for receiving metadata transmitted by the traffic device. The proxy obtains metadata from the traffic device by listening to the port, and these data contain various details of network communication, such as protocol type, source and destination information, timestamp, etc. The function of the proxy is not only to passively receive, but more importantly, to intelligently distribute the received data to the backend server cluster according to system strategies.

[0053] In the embodiment of the application, after the proxy Nginx receives, according to the current load of each server (such as the size of free memory and the CPU (Central Processing Unit) idle rate, etc.), the IP hash algorithm is used to distribute the logs from the same source device to the specified backend node (server) fixedly, so that the continuity of data is ensured, and efficient load balancing is realized. In addition, in view of the flood of metadata that may occur, the proxy also enables port multiplexing and multi-threaded receiver mechanism, improves the processing capacity of a single node, thereby reducing the possibility of data packet loss and enhancing the robustness of the system.

[0054] Then, for the metadata stored in each server memory, the first preset metadata can be obtained by preprocessing the metadata through the Spring Boot framework. This process covers standardization, cleaning and preliminary analysis of metadata to eliminate invalid or duplicate data, and at the same time, the original log data is decoded according to the preset standard (such as the RFC5424 standard (structured event message format)) to convert the unstructured metadata into the first preset metadata containing key fields. The Spring Boot framework provides a high-performance asynchronous non-blocking I / O (input / output) mechanism that can effectively handle network traffic, even in a high-concurrency environment, and can maintain good response speed and stability. In addition, the Spring Boot framework can perform message boundary detection mechanism to solve the data packet problem and ensure the integrity and accuracy of each metadata.

[0055] In this embodiment, the performance bottleneck and data quality problem of metadata receiving and preprocessing in the network security scenario are effectively solved, and efficient processing and storage of data are realized. Through reasonable configuration of the proxy and cooperative work of the backend server cluster, the system can stably cope with large-scale, different protocol network traffic data, and ensure the integrity, consistency and timeliness of the data, providing a solid data foundation for subsequent real-time analysis and decision support.

[0056] In order to improve the accuracy of preprocessing the metadata, in the traffic data monitoring method provided in Embodiment One of the application, the metadata is analyzed, and multiple groups of key-value pairs in the metadata are extracted; the multiple groups of key-value pairs are cleaned to obtain multiple groups of initial key-value pairs; the initial key-value pairs are normalized to obtain target key-value pairs; and all target key-value pairs are integrated to obtain first preset metadata.

[0057] In the embodiment of the application, due to different transmission protocols, the original metadata may be stored in different formats. A preset parser or regular expression can be used to analyze the metadata, and according to the format and structure of the metadata, multiple groups of key-value pairs containing key information are extracted to obtain structured data.

[0058] The extracted key-value pairs may contain useless or erroneous information, which can be cleaned to remove these unnecessary or malformed data, resulting in a cleaner and more effective set of initial key-value pairs. Cleaning may include deleting null values, correcting format errors (such as converting date-time fields to standard formats), filtering out irrelevant metadata (such as removing debugging information, etc.), ensuring that the data for subsequent processing is valid and structured.

[0059] After that, the initial key-value pairs are normalized to obtain target key-value pairs. Normalization is the process of converting data from different sources or formats into a unified format or structure, ensuring that all data is processed under the same standard, facilitating subsequent data aggregation, analysis, and storage. The normalization process may include converting all timestamp fields to standard time, standardizing IP address fields represented in different ways to a unified representation, mapping text description fields to predefined categories or values, etc. Through this process, the target key-value pairs obtained will be more consistent, thereby improving the reliability and efficiency of data processing. After that, all cleaned and normalized target key-value pairs are integrated into a structured data set, i.e., the first preset metadata. This data set will include all key network security information, such as source address, destination address, protocol, port, timestamp, threat level, etc., and the data format and structure have been standardized, facilitating subsequent real-time analysis by data processing frameworks (such as Flink).

[0060] In this embodiment, it can be ensured that the original data captured from the source is efficiently and accurately converted into structured and standardized metadata, thereby improving the efficiency and effectiveness of subsequent data processing. Among them, cleaning and normalization processing reduces noise and inconsistency in the data, making the data more suitable for real-time stream processing and storage. The generation of the first preset metadata provides an accurate data source for real-time threat detection and provides uniform format data for data analysis components, thereby accelerating the decision-making process and improving the visibility and response speed of the network security posture.

[0061] In order to improve the accuracy of processing the first preset metadata, in the flow data monitoring method provided by the embodiment one of the application, the field requirement is determined based on the transmission protocol corresponding to the partition, wherein the field requirement contains a preset field set, and the preset field set includes a plurality of preset fields; the first preset metadata in the partition is analyzed to obtain a plurality of primary keys and key values corresponding to the primary keys; each primary key is taken as a field, and all fields are matched with the preset field set to obtain target fields, wherein the target fields are preset fields that do not exist in the first preset metadata; the field values corresponding to the target fields are matched from a preset database; the key values corresponding to all fields and the field values corresponding to all target fields are normalized to obtain target key values corresponding to all fields and target field values corresponding to all target fields; and the second preset metadata is determined based on all fields, the target key values corresponding to all fields, all target fields and the target field values corresponding to all target fields.

[0062] In the embodiment of the application, first, the field set required for preliminary analysis is determined according to the transmission protocol corresponding to the data stored in each partition. The preset field set is defined according to the data structure in the network security field, including key fields such as timestamp (time stamp), source_ip (source IP), destination_ip (destination IP), protocol (protocol type), and port (port number), to ensure that the core information in the metadata can be captured and analyzed. The first preset metadata in each partition is preliminarily analyzed to extract a plurality of primary keys for uniquely identifying data records and key values corresponding to the primary keys. For example, for flow data, the primary keys can be a combination of source_ip, destination_ip and timestamp, to ensure the uniqueness and accuracy of the data in the time sequence. After the primary keys and their key values are analyzed, each primary key is taken as a field and compared with the preset field set. The target field refers to a field that is defined in the preset field set but does not directly appear in the first preset metadata. For each target field, the system will find or calculate the corresponding field value from the configured preset database (i.e., a database configured in advance to supplement or correct the metadata field information, which records the flow data generated by all flow devices, and according to different requirements, multiple tables are constructed, such as IP library, service information, threat intelligence library, etc.) to supplement the information not contained in the metadata, such as obtaining the IP address by querying the IP library.

[0063] Then, the key values corresponding to all fields and the field values corresponding to all target fields are normalized. Normalization is a key step to ensure data consistency and standardization, which can convert all field values and target field values into a predefined and unified format. For example, all timestamp field values are converted to standard time, all IP addresses are formatted into a unified address representation, and threat levels are mapped to a standard numerical range. After that, all fields and corresponding target key values, target fields and corresponding target field values are combined into a second preset metadata to ensure data integrity and consistency, so that subsequent real-time processing and storage operations can be based on standardized and structured data, improving the efficiency and accuracy of overall data processing.

[0064] In this embodiment, network security metadata from different transmission protocols can be efficiently processed, parsed and standardized. Not only does it solve the problem of missing fields in metadata parsing, but it also improves data readability and analysis through normalization, providing high-quality input for subsequent real-time threat detection, traffic analysis and data storage. In addition, the standardized metadata format makes data easier to share and process in a distributed architecture, further improving the scalability and flexibility of the entire system. For example, by accurately matching and supplementing fields, potential DDoS attack patterns can be more accurately identified without worrying about false positives or false negatives caused by data format or incomplete information. At the same time, due to the standardization of data, the storage system can more efficiently organize and compress data, reducing storage costs and speeding up query response time.

[0065] In order to realize the permanent storage of the second preset metadata, in the traffic data monitoring method provided in Embodiment One of the present application, after obtaining the second preset metadata, all second preset metadata are stored in a preset historical database, wherein the preset historical database is used for permanently storing the second preset metadata, and the step of storing the second preset metadata comprises: establishing a data table corresponding to each second preset metadata in the preset historical database; determining a sub-table field set based on the transmission protocol corresponding to each second preset metadata; filtering sub-table data corresponding to each sub-table field from the second preset metadata based on each sub-table field in the sub-table field set; and storing each sub-table data in a sub-table in the data table.

[0066] In the embodiment of the present application, all processed second preset metadata can be stored in a preset historical database, which is used for permanently storing the second preset metadata after cleaning and normalization.

[0067] In the embodiment of the application, a data table corresponding to each second preset metadata can be established in the preset historical database, and the creation of each data table closely corresponds to a specific type or source of the second preset metadata, ensuring data classification storage. For example, a data table can be created for metadata of the same transmission protocol, and the like. In this way, the query and analysis are simplified, and specific types of data can be located and retrieved more quickly through classified storage of data.

[0068] In the embodiment of the application, in each data table, the field set is further refined according to the transmission protocol carried by the second preset metadata to form a sub-table field set. For example, a sub-table is established according to the time field, a sub-table is established according to the IP address, and the like. Then, according to the sub-table field set, the data part corresponding to each sub-table field is filtered from the second preset metadata to form sub-table data. For example, the second preset metadata is parsed and sorted to ensure that the key information related to each sub-table is accurately separated. For example, all data related to the time field is extracted from the metadata to form a sub-table named after the time field. Then, the filtered sub-table data is stored in the sub-table of the corresponding data table in the preset historical database. Each sub-table data is inserted into the sub-table to which it belongs to maintain the structure and classification of the data.

[0069] In the embodiment, the second preset metadata is stored in a refined and structured manner, improving the efficiency of the historical database and the speed of data retrieval. The classified storage of data and the sub-table mechanism ensure that specific protocol or event type data can be quickly located and analyzed even when faced with massive data. This strategy is particularly suitable for deep threat analysis and compliance audits in network security scenarios, as it allows rapid focus on key network behaviors or abnormal events without having to traverse the entire database. In addition, the permanent storage feature of the preset historical database makes it possible to preserve and retrospectively analyze historical data, providing a solid foundation for long-term data trend analysis and in-depth research.

[0070] Optionally, each second preset metadata carries a timestamp. In order to realize real-time storage of the second preset metadata, in the traffic data monitoring method provided in Embodiment One of the application, the second preset metadata is divided into multiple shards, and a copy data of each shard is created; based on the timestamp, a primary key index of each shard is established, and based on a key field of the second preset metadata, a field index of each shard is established, wherein the key field is a field obtained by analyzing the second preset metadata, and the key field includes at least one of the following: a source address field, a threat type field; each shard associated with the primary key index and the field index is stored in multiple storage nodes in the preset real-time database; based on the timestamp, the storage duration of the shard is determined, and in the case where the storage duration of the shard reaches an effective duration, the shard is deleted from the preset real-time database.

[0071] In embodiments of the present application, the second preset metadata is divided into multiple shards according to certain rules (such as a hash function based on IP address or timestamp) before entering the preset real-time database. Each shard contains a part of the records of the overall metadata, and this division helps to evenly distribute data to multiple storage nodes, thereby achieving balanced read-write load. Subsequently, in order to ensure high availability of data, one or more replica data are created for each shard. These replica data are distributed on different storage nodes, so that even if a node fails, the same data can be read from other nodes, reducing the risk of data loss and enhancing the fault tolerance of the system.

[0072] In embodiments of the present application, the timestamp can be used as the primary key index of each shard. In each shard, all records are sorted by timestamp, establishing a primary key index. This index structure not only speeds up the query based on the time range, but also simplifies the data insertion process, because ClickHouse automatically performs data merging and sorting based on the primary key index.

[0073] In embodiments of the present application, in addition to the timestamp, field indexes based on other key fields of the metadata, such as the source address field and the threat type field, are also needed. These field indexes are used to speed up the query of specific fields, especially when fast filtering and aggregation based on source address or threat type are required. The construction of field indexes improves query efficiency and reduces the time spent on scanning large amounts of irrelevant data.

[0074] Then, each shard with primary key index and field index is stored on multiple storage nodes in the preset real-time database. The preset real-time database adopts a distributed architecture, such as a ClickHouse cluster, to achieve fast read-write and high-concurrency access of data. Data distribution and index sharing between storage nodes ensure the overall performance and reliability of the system.

[0075] In embodiments of the present application, in order to manage storage costs and data life cycle, a storage duration strategy based on timestamp is proposed. Each shard has a preset life cycle in the database, and once this period (such as 7 days) is reached, the system will automatically detect and delete the no longer needed shards from the preset real-time database. This strategy ensures that the real-time database maintains efficient access to recent data while avoiding unnecessary long-term storage costs.

[0076] In this embodiment, not only the performance and storage efficiency of the real-time database are optimized, but also the high availability and fast query capability of the system are ensured even in the case of large data volume. The use of sharded storage and multi-level indexing enables the system to complete complex multi-dimensional queries and aggregate statistics within sub-second, improving the real-time performance and effectiveness of network security monitoring. In addition, the dynamic data lifecycle management strategy balances the storage cost and data access demand, enabling the system to run for a long time without performance degradation due to data accumulation.

[0077] In order to realize real-time monitoring of the message queue, in the method for monitoring traffic data provided in Embodiment One of the present application, after each first preset metadata is sent to a different partition in the preset message queue, each partition in the preset message queue is monitored; in the case where the amount of data stored in the partition reaches a preset data amount, the partition is blocked; resource data of each partition is collected, and based on the resource data, a resource index value of the partition is determined; in the case where the resource index value does not belong to a preset index range, the partition is blocked.

[0078] In the embodiment of the present application, a monitoring component can be built, which is responsible for continuously monitoring each partition in the Kafka message queue. Kafka, as a high-performance distributed message system, one of its core features is the ability to subdivide data streams into multiple partitions, each of which can be stored and processed individually. By monitoring the status of each partition, including but not limited to data volume, number of unconsumed messages, activity of consumer groups, etc., potential problems such as data backlog or resource bottlenecks can be discovered and handled in a timely manner. When the monitoring component detects that the data volume of a certain partition exceeds the preset threshold, the partition will be temporarily blocked. The preset data volume is a threshold set according to system resources and processing capacity to avoid a single partition storing too many messages, leading to rising processing delays or intensified resource contention. Once the partition is blocked, the system will pause adding new messages to the partition until its data volume drops to a safe level, which helps to maintain the stability and responsiveness of the entire message system.

[0079] In addition, the monitoring component can periodically collect resource data of each partition, including CPU usage, memory occupancy, disk I / O, network bandwidth and other key performance indicators. Based on these resource data, the system calculates the resource indicator value of each partition, which is a quantitative standard for measuring the degree of resource consumption of the partition. Continuous monitoring of the resource indicator value can understand the health status of the message queue and predict possible performance bottlenecks. If the resource indicator value of the partition exceeds the preset indicator range, the monitoring component will implement the blocking operation on the partition. The preset indicator range is a set of threshold rules based on system performance and stability requirements, including the upper and lower limits of resource consumption. Partitions exceeding the indicator range are considered high risk and may affect the overall system efficiency. The blocking operation can prevent further excessive resource consumption and trigger subsequent troubleshooting or resource allocation processes.

[0080] In this embodiment, not only can resource bottlenecks be prevented and timely responded to, ensuring real-time and accuracy of data processing, but also the overall performance and reliability of the system can be improved through dynamic partition management. That is, by blocking the partitions that reach the preset data volume or resource indicator abnormality, data processing delays can be avoided, the risk of message loss can be reduced, and reasonable allocation of resources can be ensured, avoiding system instability caused by a single hotspot.

[0081] Figure 2 is a schematic diagram of a data monitoring process using a data monitoring system according to an embodiment of the present application, as shown in Figure 2As shown, the data monitoring system includes: a load balancing processing framework, a big data processing computing cluster, a big data storage cluster of distributed architecture, a data analysis / monitoring component. Among them, the full-flow probe can collect the flow data generated by the full-flow device. Different flow devices use different transmission protocols, such as Ftp protocol, Database protocol, smtp protocol, udp protocol, telnet protocol, Dns protocol, http protocol, etc. The full-flow device can export the respective flow data to the load balancing processing framework. The load balancing processing framework includes: a master Nginx and a backup Nginx, which can be switched to the backup Nginx when the master Nginx fails. The Nginx can store all the received data to different servers Syslog (a standardized log message format and communication protocol), and the SpringBoot collection service runs on each server, responsible for preliminary data collection, through port listening, message decoding, key value extraction, data generalization and data normalization, etc. To get structured data, and send the structured data to the big data processing computing cluster. The big data processing computing cluster can store data in different partitions in the Kafka message cluster according to different transmission protocols, for example, Ftp protocol data is stored in Topic (partition) A, Smtp protocol data is stored in Topic B, Udp protocol data is stored in Topic C, Telnet protocol data is stored in Topic D, Http protocol data is stored in Topic E. Then, Flink can be used to parse, filter, correct, convert and enhance the data in each partition. During the Flink processing, data can be read from mysql (a database) (including: IP library, business information table, etc.) to supplement the fields. After that, the processed data is persisted to the big data storage cluster of distributed architecture. The Hive (preset historical database) in the big data storage cluster of distributed architecture is used for persistent storage of data, supporting batch analysis and offline processing of data. The Hive can include: a source layer, a detail layer, a dimension layer, a summary layer, and an application layer. Clickhouse in the big data storage cluster of distributed architecture is used to store real-time data and record data distribution through the construction of a distributed table. In addition, in the data analysis / monitoring component, Kafka-Eagle (a monitoring tool) can be used to monitor the health status of the Kafka queue in real time, including cluster health monitoring, security audit, automatic alarm, etc. Metabase can be associated with Clickhouse to monitor the data in the database in real time, and perform multi-dimensional monitoring such as web application attacks, abnormal process behavior, abnormal data transmission, attack flow characteristics, threat intelligence correlation and attack attempts.

[0082] In the embodiment of the present application, through the load balancing processing framework in the data monitoring system, the metadata transmitted by different traffic devices can be efficiently load balanced and processed, and each first preset metadata after load balancing processing is stored in different partitions of a preset message queue in a big data processing computing cluster in the data monitoring system. The first preset metadata of each partition is processed by the big data processing computing cluster, and the second preset metadata after processing is stored in a preset real-time database in a big data storage cluster in the data monitoring system, so as to realize real-time and visual monitoring of traffic data, achieve the purpose of efficient processing and storage of metadata transmitted by different traffic devices, thereby realizing the technical effect of real-time analysis and monitoring of network security metadata, and further solving the technical problem of low efficiency of monitoring a large amount of traffic data in related technologies.

[0083] The following will be described in detail in combination with another embodiment.

[0084] Embodiment two

[0085] The traffic data monitoring device provided in the embodiment comprises a plurality of implementation units, each of which corresponds to each implementation step in the above embodiment one.

[0086] Figure 3 is a schematic diagram of an optional traffic data monitoring device according to the embodiment of the present application, as shown in Figure 3 The monitoring device can include a receiving unit 30, a first processing unit 31, a second processing unit 32, and a monitoring unit 33.

[0087] The receiving unit 30 is configured to receive metadata transmitted by different traffic devices, wherein the transmission protocols used by different traffic devices are different, and the metadata is traffic data transmitted by the traffic devices based on the transmission protocols.

[0088] The first processing unit 31 is configured to perform load balancing processing on all received metadata to obtain first preset metadata corresponding to each metadata, and send each first preset metadata to different partitions in a preset message queue based on the transmission protocol corresponding to each first preset metadata.

[0089] The second processing unit 32 is configured to process the first preset metadata of each partition to obtain second preset metadata, and store all second preset metadata in a preset real-time database, wherein the second preset metadata is metadata that meets the field requirements of the corresponding transmission protocol.

[0090] The monitoring unit 33 is configured to monitor all second preset metadata in the preset real-time database, and visually display the monitoring results.

[0091] The monitoring device can perform efficient load balancing processing on the metadata transmitted by different traffic equipment through a load balancing processing framework in the data monitoring system, store each first preset metadata after load balancing processing to different partitions in a preset message queue in a big data processing computing cluster in the data monitoring system, process the first preset metadata in each partition through the big data processing computing cluster, and store the second preset metadata after processing to a preset real-time database in a big data storage cluster in the data monitoring system, so as to realize real-time and visual monitoring of traffic data, achieve the purpose of efficient processing and storage of metadata transmitted by different traffic equipment, thereby realizing the technical effects of real-time analysis and monitoring of network security metadata, and further solving the technical problem of low efficiency of monitoring a large amount of traffic data in the related art.

[0092] Optionally, the data monitoring system comprises at least a proxy and a plurality of servers, and the first processing unit comprises a first storage module configured to store all the metadata to the plurality of servers based on the current load of each server if the proxy receives the plurality of metadata, wherein the proxy acquires the metadata by listening to a preset port, and the preset port is a port opened by the data monitoring system to the traffic equipment; and a first processing module configured to pre-process the metadata stored in each server to obtain the first preset metadata.

[0093] Optionally, the first processing module comprises a first analysis submodule configured to analyze the metadata and extract a plurality of groups of key-value pairs in the metadata; a first cleaning submodule configured to clean the plurality of groups of key-value pairs to obtain a plurality of groups of initial key-value pairs; a first normalization submodule configured to normalize the initial key-value pairs to obtain target key-value pairs; and a first integration submodule configured to integrate all the target key-value pairs to obtain the first preset metadata.

[0094] Optionally, the second processing unit comprises: a first determining module configured to determine a field requirement based on a transmission protocol corresponding to the partition, wherein the field requirement comprises a preset field set, and the preset field set comprises a plurality of preset fields; a first parsing module configured to parse the first preset metadata in the partition to obtain a plurality of primary keys and key values corresponding to the primary keys; a first matching module configured to take each primary key as a field, and match all the fields with the preset field set to obtain target fields, wherein the target fields are preset fields that do not exist in the first preset metadata; a second matching module configured to match field values corresponding to the target fields from the preset database; a second processing module configured to perform normalization processing on the key values corresponding to all the fields and the field values corresponding to all the target fields to obtain target key values corresponding to all the fields and target field values corresponding to all the target fields; and a second determining module configured to determine the second preset metadata based on all the fields, the target key values corresponding to all the fields, all the target fields, and the target field values corresponding to all the target fields.

[0095] Optionally, the monitoring device further comprises a second storage module configured to store all the second preset metadata to a preset historical database after obtaining the second preset metadata, wherein the preset historical database is configured to permanently store the second preset metadata, and the step of storing the second preset metadata comprises: establishing a data table corresponding to each second preset metadata in the preset historical database; determining a sub-table field set based on a transmission protocol corresponding to each second preset metadata; screening sub-table data corresponding to each sub-table field from the second preset metadata based on each sub-table field in the sub-table field set; and storing each sub-table data in a sub-table in the data table.

[0096] Optionally, each second preset metadata carries a timestamp, and the second processing unit further comprises: a first division module configured to divide the second preset metadata into a plurality of shards, and create replica data of each shard; a first establishment module configured to establish a primary key index of each shard based on the timestamp, and establish a field index of each shard based on a key field of the second preset metadata, wherein the key field is a field obtained by analyzing the second preset metadata, and the key field comprises at least one of the following: a source address field, a threat type field; a third storage module configured to store each shard associated with the primary key index and the field index to a plurality of storage nodes in a preset real-time database; and a third determining module configured to determine a storage duration of the shard based on the timestamp, and delete the shard from the preset real-time database when it is detected that the storage duration of the shard reaches an effective duration.

[0097] Optionally, the monitoring device further comprises: a first monitoring module, configured to monitor each partition in the preset message queue after each first preset metadata is sent to a different partition in the preset message queue; a first blocking module, configured to block the partition if the amount of data stored in the partition reaches a preset data amount; a first collection module, configured to collect resource data of each partition, and determine a resource index value of the partition based on the resource data; and a second blocking module, configured to block the partition if the resource index value does not belong to a preset index range.

[0098] The monitoring device described above can further comprise a processor and a memory, and the receiving unit 30, the first processing unit 31, the second processing unit 32, the monitoring unit 33 and the like are stored in the memory as program units, and the corresponding functions are realized by the processor executing the program units stored in the memory.

[0099] The processor described above comprises a kernel, and the kernel retrieves the corresponding program units from the memory. The kernel can be one or more, and the kernel parameters are adjusted to monitor all second preset metadata in the preset real-time database, and the monitoring results are visually displayed.

[0100] The memory described above can comprise a non-permanent memory in a computer readable medium, a random access memory (RAM) and / or a non-volatile memory such as a read-only memory (ROM) or a flash memory (flash RAM), and the memory comprises at least one memory chip.

[0101] The application further provides a computer program product adapted to execute the program of the following method steps when executed on a data processing device: receiving metadata transmitted by different traffic devices, performing load balancing processing on all received metadata to obtain first preset metadata corresponding to each metadata, sending each first preset metadata to a different partition in a preset message queue based on a transmission protocol corresponding to each first preset metadata, processing the first preset metadata of each partition to obtain second preset metadata, storing all second preset metadata to a preset real-time database, monitoring all second preset metadata in the preset real-time database, and visually displaying the monitoring results.

[0102] According to another aspect of the embodiments of the application, a computer program product is also provided, comprising a non-volatile computer readable storage medium, the non-volatile computer readable storage medium storing a computer program, and the computer program is executed by a processor to implement the traffic data monitoring method of any one of the above.

[0103] According to another aspect of the embodiments of the present application, an electronic device is also provided, which includes one or more processors and a memory storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the above-mentioned method for monitoring traffic data.

[0104] Figure 4 is a hardware structure block diagram of an electronic device (or a mobile device) for a method for monitoring traffic data according to an embodiment of the present application. As shown in Figure 4 , the electronic device can include one or more processors (for example, processors 402a, 402b, …, 402n, etc. in Figure 4 , which can include but are not limited to processing devices such as microprocessors MCU or programmable logic devices FPGA, etc.), a memory 404 for storing data. In addition, it can also include a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which can be included as one of the ports of the I / O interface), a network interface, a keyboard, a power supply and / or a camera. Those skilled in the art can understand that Figure 4 The structure shown is only schematic, which does not limit the structure of the above-mentioned electronic device. For example, the electronic device can also include more or fewer components than those shown in Figure 4 , or have a different configuration from Figure 4 .

[0105] The above-mentioned serial numbers of the embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0106] The embodiments or examples of the present disclosure are not exhaustive, but only a part of the embodiments or examples, and are not specific limitations on the protection scope of the present disclosure. In the case of no contradiction, each step in an embodiment or example can be implemented as an independent embodiment, and the steps can be combined arbitrarily, for example, the scheme after removing part of the steps in an embodiment or example can also be implemented as an independent embodiment, and the order of the steps in an embodiment or example can be exchanged arbitrarily, in addition, the optional ways or optional examples in an embodiment or example can be combined arbitrarily; in addition, the embodiments or examples can be combined arbitrarily, for example, the steps of different embodiments or examples can be combined arbitrarily, an embodiment or example can be combined with the optional ways or optional examples of other embodiments or examples.

[0107] In the above-mentioned embodiments of the present application, the description of each embodiment has its own emphasis, and the parts not described in detail in an embodiment can be referred to the related description of other embodiments.

[0108] In several embodiments provided by the present application, it should be understood that the disclosed technology can be implemented in other manners. For example, the described unit embodiments can be divided into other ways, for example, the units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be implemented by using some interfaces, and the indirect couplings or communication connections can be implemented in electronic, mechanical, or other forms.

[0109] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, can be located in one place, or can be distributed on multiple units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0110] In addition, each functional unit in the various embodiments of the present application can be integrated into a processing unit, or each unit can exist physically, or two or more units can be integrated into one unit. The integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0111] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The foregoing storage medium includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various program codes that can be stored in the medium.

[0112] The above description is only the preferred embodiment of the present application, and it should be pointed out that for ordinary skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, and these improvements and refinements should be considered as the protection scope of the present application.

Claims

1. A method of monitoring traffic data, characterized by, The application is applied to a data monitoring system, comprising: receiving metadata transmitted by different flow devices, wherein the different flow devices use different transmission protocols, and the metadata is flow data transmitted by the flow devices based on the transmission protocols; performing load balancing processing on all the received metadata to obtain first preset metadata corresponding to each of the metadata, and sending each of the first preset metadata to different partitions in a preset message queue based on the transmission protocol corresponding to each of the first preset metadata; processing the first preset metadata in each of the partitions to obtain second preset metadata, and storing all the second preset metadata to a preset real-time database, wherein the second preset metadata is metadata meeting the field requirements corresponding to the transmission protocol; monitoring all the second preset metadata in the preset real-time database and visually displaying the monitoring results.

2. The monitoring method according to claim 1, characterized in that, The data monitoring system at least comprises an agent and multiple servers, and the step of performing load balancing processing on all the received metadata to obtain first preset metadata corresponding to each of the metadata comprises: in the case that the agent receives multiple metadata, storing all the metadata to multiple servers based on the current load of each server, wherein the agent obtains the metadata by listening to a preset port, and the preset port is a port opened by the data monitoring system to the flow devices; performing preprocessing on the metadata stored in each of the servers to obtain the first preset metadata.

3. The monitoring method according to claim 2, characterized in that, The step of performing preprocessing on the metadata stored in each of the servers to obtain the first preset metadata comprises: parsing the metadata to extract multiple groups of key-value pairs in the metadata; cleaning the multiple groups of key-value pairs to obtain multiple groups of initial key-value pairs; normalizing the initial key-value pairs to obtain target key-value pairs; integrating all the target key-value pairs to obtain the first preset metadata.

4. The monitoring method of claim 1, wherein, The step of processing the first preset metadata in each of the partitions to obtain second preset metadata comprises: determining the field requirements based on the transmission protocol corresponding to the partition, wherein the field requirements contain a preset field set, and the preset field set comprises multiple preset fields; parsing the first preset metadata in the partition to obtain multiple primary keys and key values corresponding to the primary keys; taking each of the primary keys as a field, matching all the fields with the preset field set to obtain target fields, wherein the target fields are the preset fields that do not exist in the first preset metadata; matching field values corresponding to the target fields from a preset database; performing normalization processing on the key values corresponding to all the fields and the field values corresponding to all the target fields to obtain target key values corresponding to all the fields and target field values corresponding to all the target fields; Determine the second preset metadata based on all the fields, the target key values corresponding to all the fields, all the target fields, and the target field values corresponding to all the target fields.

5. The monitoring method of claim 1, wherein, After obtaining the second preset metadata, further comprising: Store all the second preset metadata to a preset historical database, wherein the preset historical database is used to permanently store the second preset metadata, and the step of storing the second preset metadata comprises: Establish a data table corresponding to each of the second preset metadata in the preset historical database; Determine a sub-table field set based on the transmission protocol corresponding to each of the second preset metadata; Filter sub-table data corresponding to each of the sub-table fields from the second preset metadata based on each of the sub-table fields in the sub-table field set; Store each of the sub-table data in a sub-table in the data table.

6. The monitoring method of claim 1, wherein, Each of the second preset metadata carries a timestamp, and the step of storing all the second preset metadata to a preset real-time database comprises: Divide the second preset metadata into multiple shards and create replica data for each of the shards; Establish a primary key index for each of the shards based on the timestamp, and establish a field index for each of the shards based on a key field of the second preset metadata, wherein the key field is a field obtained by analyzing the second preset metadata, and the key field comprises at least one of the following: a source address field, a threat type field; Store each of the shards associated with the primary key index and the field index to multiple storage nodes in the preset real-time database; Determine a storage duration of the shard based on the timestamp, and delete the shard from the preset real-time database if the storage duration of the shard reaches a valid duration.

7. The monitoring method of claim 1, wherein, After each of the first preset metadata is sent to different partitions in a preset message queue, further comprising: Monitor each of the partitions in the preset message queue; Block the partition if the amount of data stored in the partition reaches a preset data amount; Collect resource data of each of the partitions, and determine a resource indicator value of the partition based on the resource data; Block the partition if the resource indicator value does not belong to a preset indicator range.

8. A monitoring apparatus of traffic data, characterized by, Applied to a data monitoring system, comprising: A receiving unit configured to receive metadata transmitted by different traffic devices, wherein the transmission protocols used by the different traffic devices are different, and the metadata is traffic data transmitted by the traffic devices based on the transmission protocols; A first processing unit configured to perform load balancing processing on all the received metadata to obtain first preset metadata corresponding to each of the metadata, and send each of the first preset metadata to different partitions in a preset message queue based on the transmission protocol corresponding to each of the first preset metadata. A second processing unit is configured to process the first preset metadata of each of the partitions to obtain second preset metadata, and store all the second preset metadata into a preset real-time database, wherein the second preset metadata is metadata satisfying a field requirement corresponding to the transmission protocol; A monitoring unit is configured to monitor all the second preset metadata in the preset real-time database, and visually display a monitoring result.

9. A computer program product, characterised in that, A non-volatile computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the traffic data monitoring method in any one of claims 1 to 7.

10. An electronic device, comprising: One or more processors and a memory are included, and the memory is configured to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the traffic data monitoring method in any one of claims 1 to 7.

Images