Traffic monitoring method, system and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a traffic monitoring method, system and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: for each target cloud native network application firewall WAF node, respectively obtaining traffic characteristic data of the target cloud native WAF node; converging and storing traffic feature data of the target cloud native WAF nodes; according to a preset time dimension, counting traffic characteristic data of each target cloud native WAF node under each traffic characteristic data type dimension, and storing a statistical result to a target database; constructing a traffic monitoring view according to the data stored in the target database, a predetermined traffic monitoring dimension and a preset traffic monitoring view construction rule; the flow monitoring view is used for monitoring the cloud native WAF system. By adopting the method, traffic monitoring can be carried out on the cloud native WAF system.

Description

Technical Field

[0001] The present application relates to the field of cloud computing technology, and in particular to a traffic monitoring method, system, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] At present, with the rapid development of cloud computing technology, more and more companies are migrating their businesses to cloud platforms. The resulting network security threats are also increasing, especially attacks at the application level such as Structured Query Language (SQL) injection and cross-site scripting, which seriously threaten the data security of enterprises. Web Application Firewall (WAF) is a specific form of application firewall used to filter, monitor and block Hypertext Transfer Protocol (HTTP) traffic through web services. By monitoring HTTP traffic, it can prevent attacks that exploit known vulnerabilities in web applications, such as SQL injection and cross-site scripting (XSS). Therefore, cloud-native WAF systems have been widely used.

[0003] Faced with massive network traffic, how to monitor and analyze traffic and detect abnormal situations in time to better operate cloud-native WAF systems and serve customers has become an urgent problem that cloud-native WAF systems need to solve. Therefore, a traffic monitoring method that can monitor traffic in cloud-native WAF systems is urgently needed. Summary of the invention

[0004] Based on this, it is necessary to provide a traffic monitoring method, system, device, computer equipment, computer-readable storage medium and computer program product that can monitor the traffic of a cloud-native WAF system in response to the above technical problems.

[0005] In a first aspect, the present application provides a flow monitoring method, comprising:

[0006] For each target cloud native web application firewall WAF node, obtain the traffic characteristic data of the target cloud native WAF node respectively;

[0007] Aggregate and store traffic characteristic data of each target cloud native WAF node;

[0008] According to the preset time dimension, the traffic characteristic data of each target cloud native WAF node under each traffic characteristic data type dimension is counted, and the statistical results are stored in the target database;

[0009] A traffic monitoring view is constructed based on the data stored in the target database, predetermined traffic monitoring dimensions and preset traffic monitoring view construction rules; the traffic monitoring view is used to monitor the cloud native WAF system.

[0010] In one of the embodiments, counting the traffic characteristic data of each target cloud native WAF node under each traffic characteristic data type dimension according to a preset time dimension, and storing the statistical results in a target database, includes:

[0011] According to the current business volume of each target cloud native WAF node, multiple target consumers are enabled;

[0012] For each target consumer, the traffic characteristic data received by the target consumer is written into the channel of the target consumer corresponding to each traffic characteristic data type dimension through the target consumer;

[0013] For each traffic feature data type dimension, through the coroutine of the target consumer corresponding to the traffic feature data type dimension, according to the preset time dimension, statistics are collected on the traffic feature data in the channel of the target consumer corresponding to the traffic feature data type dimension;

[0014] Writing the traffic feature data after statistics into the write channel of the target consumer corresponding to the traffic feature data type dimension;

[0015] If the preset database writing condition is met, the statistically analyzed traffic characteristic data in the writing channel is written into the target database.

[0016] In one embodiment, the method further comprises:

[0017] When a preset first time node is reached, the target historical data of the cloud native WAF system is counted and recorded in a temporary file; the target historical data includes the historical request volume, the historical attack volume and the historical quantity of each status code; and a first time period is separated between two adjacent first time nodes;

[0018] When the preset second time node is reached, the target historical data between the previous second time node and the current second time node stored in the temporary file is written into the target database; a second time period is separated from two adjacent second time nodes; and the length of the second time period is an integer multiple of the length of the first time period.

[0019] In one embodiment, writing the target historical data between the last second time node and the current second time node stored in the temporary file into the target database includes:

[0020] If a new status code is added when writing to the target database, modify the database table field corresponding to the status code according to the range of the status code and the new status code;

[0021] Based on the modified database table field, split the historical number of each status code between the previous second time node and the current second time node stored in the temporary file into a preset number of status code data tables;

[0022] Write each of the status code data tables and the total request data table into the target database.

[0023] In one embodiment, the method further comprises:

[0024] If the current time and the last recording time of the temporary file are in different first time periods and in the same second time period, the time period between the current time and the last recording time is used as the first missing time period;

[0025] The target historical data of the cloud native WAF system in the first missing time period is counted, and the target historical data in the first missing time period is recorded in the temporary file.

[0026] In one embodiment, the method further comprises:

[0027] If the current time and the last recording time of the temporary file are in different second time periods, obtaining the earliest retention time of the target historical data retained in the target database;

[0028] The latest time between the last recording time and the earliest retention time is used as the starting time for statistics;

[0029] The time period between the statistical start time and the current time is taken as the second missing time period. Starting from the statistical start time, according to the first time node and the second time node, statistical processing is performed on the target historical data of the cloud native WAF system within the second missing time period.

[0030] In a second aspect, the present application also provides a flow monitoring system, including:

[0031] At least one distributed data collection module, used for collecting traffic characteristic data of each target cloud native network application firewall WAF node respectively; each of the target cloud native WAF nodes includes one distributed data collection module;

[0032] A data aggregation center, used to aggregate and store traffic characteristic data of each target cloud native WAF node;

[0033] A distributed data processing module is used to count the traffic characteristic data of each target cloud native WAF node under each traffic characteristic data type dimension according to a preset time dimension, and store the statistical results in a target database;

[0034] The storage service module is used to build a traffic monitoring view based on the data stored in the target database, predetermined traffic monitoring dimensions and preset traffic monitoring view building rules; the traffic monitoring view is used to monitor the cloud native WAF system.

[0035] In a third aspect, the present application also provides a flow monitoring device, comprising:

[0036] The first acquisition module is used to respectively acquire the traffic characteristic data of each target cloud native network application firewall WAF node;

[0037] An aggregation module, used to aggregate and store traffic characteristic data of each target cloud native WAF node;

[0038] A first statistical module is used to count the traffic characteristic data of each target cloud native WAF node under each traffic characteristic data type dimension according to a preset time dimension, and store the statistical results in a target database;

[0039] A construction module is used to construct a traffic monitoring view based on the data stored in the target database, predetermined traffic monitoring dimensions and preset traffic monitoring view construction rules; the traffic monitoring view is used to monitor the cloud native WAF system.

[0040] In a fourth aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps described in the first aspect when executing the computer program.

[0041] In a fifth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps described in the first aspect are implemented.

[0042] In a sixth aspect, the present application further provides a computer program product, comprising a computer program, which implements the steps described in the first aspect when executed by a processor.

[0043] The above-mentioned traffic monitoring method, system, device, computer equipment, computer-readable storage medium and computer program product respectively obtain the traffic characteristic data of each target cloud native network application firewall WAF node for each target cloud native WAF node; aggregate and store the traffic characteristic data of each target cloud native WAF node; according to the preset time dimension, count the traffic characteristic data of each target cloud native WAF node under each traffic characteristic data type dimension, and store the statistical results to the target database; construct a traffic monitoring view according to the data stored in the target database, the predetermined traffic monitoring dimension and the preset traffic monitoring view construction rules; the traffic monitoring view is used to monitor the cloud native WAF system. In this way, the traffic characteristic data of each target cloud native WAF node is respectively obtained, then aggregated, stored, and counted according to the preset time dimension and each traffic characteristic data type dimension, so as to construct a traffic monitoring view for monitoring the cloud native WAF system, which can realize traffic monitoring of the cloud native WAF system. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0045] FIG1 is a schematic diagram of a flow monitoring system according to an embodiment;

[0046] FIG2 is a schematic diagram of a flow chart of a flow monitoring method in one embodiment;

[0047] 3 is a flowchart of the steps of counting traffic feature data of each target cloud native WAF node under each traffic feature data type dimension according to a preset time dimension, and storing the statistical results in a target database in one embodiment;

[0048] FIG4 is a schematic diagram of the distributed data processing module 130 using multiple consumers and multiple coroutines to process traffic feature data;

[0049] FIG5 is a flow chart showing steps further included in the flow monitoring method in one embodiment;

[0050] 6 is a flowchart of a step of writing target historical data between the last second time node and the current second time node stored in a temporary file into a target database in one embodiment;

[0051] FIG7 is a flow chart showing steps further included in the flow monitoring method in another embodiment;

[0052] FIG8 is a flow chart showing steps further included in the flow monitoring method in another embodiment;

[0053] FIG9 is a structural block diagram of a flow monitoring device in one embodiment;

[0054] FIG. 10 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] The flow monitoring method provided in the embodiment of the present application can be applied to the flow monitoring system 100 shown in FIG1 . The flow monitoring system 100 includes: at least one distributed data acquisition module 110, a data aggregation center 120, a distributed data processing module 130 and a storage service module 140. Among them:

[0057] At least one distributed data collection module 110, for collecting traffic characteristic data of each target cloud native network application firewall WAF node; each target cloud native WAF node includes a distributed data collection module;

[0058] The data aggregation center 120 is used to aggregate and store the traffic characteristic data of each target cloud native WAF node;

[0059] The distributed data processing module 130 is used to count the traffic characteristic data of each target cloud native WAF node under each traffic characteristic data type dimension according to a preset time dimension, and store the statistical results in a target database;

[0060] The storage service module 140 is used to build a traffic monitoring view based on the data stored in the target database, predetermined traffic monitoring dimensions and preset traffic monitoring view building rules; the traffic monitoring view is used to monitor the cloud native WAF system.

[0061] In the embodiment of the present application, the traffic monitoring system 100 adopts a distributed architecture of a data processing center and multiple edge collection nodes, and the data collection process and the WAF business processing process are located in the same WAF node. This allows for smooth expansion of monitoring according to the business expansion of WAF products, which can enhance scalability; monitoring begins when the WAF node starts processing business, improving real-time monitoring operation efficiency.

[0062] The distributed data acquisition module 110 and the distributed data processing module 130 can both be implemented using golang, and compiled into executable files after development, independent of the operating environment. In this way, the logic code of the traffic monitoring system is all written in golang, which makes it easier to support multiple underlying architectures, such as x86 and arm architectures. It is only necessary to specify the corresponding architecture version during compilation, which meets the requirements for WAF deployment under multiple architectures.

[0063] The distributed data collection module 110 uses multiple different coroutines to periodically acquire, process and report the traffic feature information such as tenant ID, domain name, source IP, request time, request size, request processing time, status code, etc. in the access log and attack log generated by the current target cloud native WAF node.

[0064] The data aggregation center 120 can be a remote dictionary server (Remote Dictionary Server, redis). Redis is a high-performance data storage device. It can support multiple consumers to process different data at the same time while processing the collected data in sequence by using the stream feature, and supports strong data processing capabilities. The access data and attack data can be separated by setting different consumer groups and streams, waiting for the distributed data processing module 130 to consume and process the aggregated data.

[0065] The distributed data processing module 130 supports multiple consumers to process data at the same time. This can speed up data processing and prevent data congestion in the data aggregation center. The distributed data processing module 130 enables multiple coroutines, and each consumer uses different coroutines to process each type of monitoring dimension concurrently. This can improve processing efficiency.

[0066] The storage service module 140 may provide a query service interface. The query service interface is used for data display and analysis. Users may query the traffic monitoring view under each traffic monitoring dimension through the query service interface.

[0067] In one example, the process resource usage of the running distributed data collection module 110 is limited by systemd (System daemon) in terms of the number of CPUs, memory, and file handles used. In this way, by limiting the resource usage of the distributed data collection module process CPU, memory, and file handles by systemd, the WAF product service will not be affected by the monitoring service even in extreme cases.

[0068] It is understandable that the method can also be applied to a terminal or a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. Among them, the terminal can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0069] In an exemplary embodiment, as shown in FIG. 2 , a flow monitoring method is provided, and the method is described by taking the method applied to the flow monitoring system 100 in FIG. 1 as an example. The method includes the following steps:

[0070] Step 201, for each target cloud native web application firewall WAF node, obtain the traffic characteristic data of the target cloud native WAF node respectively.

[0071] In an embodiment of the present application, a cloud native web application firewall (WAF) system includes at least one target cloud native web application firewall WAF node. The cloud native WAF system may include multiple target cloud native WAF nodes. The cloud native WAF system is a system to be monitored for traffic. The target cloud native WAF node is a cloud native WAF node to be monitored for traffic. The cloud native WAF node performs WAF business processing. The traffic characteristic data is used to calculate the traffic characteristics of the cloud native WAF system, which may include at least one of a tenant identity document (ID), a domain name, a source Internet Protocol Address (IP), a request time, a request size, a request processing time, and a status code, and may also include at least one of a request volume, an attack volume, a request length, and a response length.

[0072] For each target cloud native network application firewall WAF node, the distributed data collection module 110 included in the target cloud native WAF node collects the traffic log of the target cloud native WAF node. Then, the distributed data collection module 110 parses the traffic log to obtain the traffic feature data of the target cloud native WAF node. Then, the distributed data collection module 110 sends the traffic feature data to the data convergence center 120. Among them, the traffic log is the traffic log generated by the WAF engine detection, including access logs and attack logs.

[0073] In one example, the distributed data collection module 110 reads and processes line by line, respectively using different regular expressions to extract traffic feature information from the access log and the attack log. If the log format meets the preset format conditions, the distributed data collection module 110 sends the traffic feature information in the log to the data aggregation center 120.

[0074] Step 202: Aggregate and store traffic feature data of each target cloud native WAF node.

[0075] In the embodiment of the present application, the data aggregation center 120 aggregates the traffic feature data of each target cloud native WAF node. Then, the data aggregation center 120 uses a remote dictionary server (redis) to store the traffic feature data of each target cloud native WAF node, and uses different streams to distinguish the traffic feature data in the access log and the traffic feature data in the attack log.

[0076] In one example, the data convergence center 120 may also mount the traffic characteristic data to the host machine, so as to avoid data loss.

[0077] Step 203: According to the preset time dimension, the traffic feature data of each target cloud native WAF node under each traffic feature data type dimension is counted, and the statistical results are stored in the target database.

[0078] In an embodiment of the present application, the distributed data processing module 130 receives traffic feature data in access logs and traffic feature data in attack logs from different streams. Then, the distributed data processing module 130 counts the traffic feature data of each target cloud native WAF node under each traffic feature data type dimension according to the preset time dimension, and stores the statistical results in the target database. Among them, the time dimension can be one or more, and can include at least one of the second level, minute level and ten-minute level. The traffic feature data type is the type of traffic feature data, and the above-mentioned tenant ID, domain name, source IP, request time, request size, request processing time, status code, request volume, attack volume, request length and response length are all traffic feature data types. The traffic feature data type dimension is the dimension of the traffic feature data type. For example, the traffic feature data type dimension can include the tenant ID dimension, domain name dimension, source IP dimension, request time dimension, request size dimension, request processing time dimension and status code dimension. The target database is a database that stores the traffic feature information after statistics. For example, the target database is a postgresql database.

[0079] Step 204, constructing a traffic monitoring view according to the data stored in the target database, the predetermined traffic monitoring dimensions and the preset traffic monitoring view construction rules.

[0080] Among them, the traffic monitoring view is used to monitor the cloud native WAF system.

[0081] In an embodiment of the present application, the storage service module 140 constructs a traffic monitoring view according to the data stored in the target database, the predetermined traffic monitoring dimension and the preset traffic monitoring view construction rules. Among them, the data stored in the target database includes statistical results and may also include other data. The traffic monitoring dimension may include at least one of the time dimension and the data type dimension of each traffic feature, and may also include other dimensions. Other dimensions may be historical data dimensions. The traffic monitoring view may include real-time sub-traffic monitoring views of dimensions such as request volume, attack volume, request length, response length, request delay, and attack delay during a preset query period under dimensions such as tenants, domain names, source IPs, and status codes, and may also include historical sub-traffic monitoring views of dimensions such as historical request volume, historical attack volume, and historical data of each status code. The traffic monitoring view may include sub-traffic monitoring views of tenants, domain names, and source IPs with the TOP10 visits, sub-traffic monitoring views of the TOP10 attack volume, request size, request time, and status code dimensions, and at least one of the sub-traffic monitoring views of the historical processing request volume, attack volume, and status code of the cloud native WAF system.

[0082] In one embodiment, the request delay and the attack delay are average values. For example, the request delay is the average value of the request processing time, which can be expressed as:

[0083]

[0084] Among them, Req_avg_delay is the request delay, The current request processing time. The number of requests.

[0085] In the above traffic monitoring method, the traffic feature data of each target cloud native WAF node is obtained respectively, and then aggregated, stored, and counted according to the preset time dimension and each traffic feature data type dimension, so as to construct a traffic monitoring view for monitoring the cloud native WAF system, which can realize traffic monitoring of the cloud native WAF system. Not only that, this method can perform real-time multi-dimensional traffic monitoring for cloud native WAF products, and realize a more targeted real-time multi-dimensional traffic monitoring system for dimensions such as tenants, domain names, source IPs, and response status codes that cloud native WAF products need to pay attention to, helping WAF to better monitor the traffic conditions of each tenant, domain name, and source IP in real time, so as to better operate, provide users with better and more specific WAF usage suggestions, and provide a method for dealing with complex attacks and real-time monitoring of network traffic problems. Moreover, this method adopts a distributed collection node architecture with low resource usage; it only sends a small amount of unique data for cloud-native WAF product monitoring and reporting, such as tenant ID, domain name, source IP, etc.; the central processing and analysis is isolated from the WAF node and does not affect the WAF core forwarding detection logic; through the collection and analysis of WAF logs, operations and R&D personnel can monitor the current status of each user, domain name, IP, and status code of the WAF, so as to help WAF products better operate and discover problems.

[0086] In an exemplary embodiment, as shown in FIG3 , according to a preset time dimension, the traffic feature data of each target cloud native WAF node under each traffic feature data type dimension is counted, and the specific process of storing the statistical results in the target database includes the following steps:

[0087] Step 301: Enable multiple target consumers according to the current business volume of each target cloud native WAF node.

[0088] In the embodiment of the present application, the distributed data processing module 130 determines the number of target consumers according to the current business volume of each target cloud native WAF node, and enables the target consumers of the same number. The target consumers are consumers that are enabled to process data, and are used to process data in parallel.

[0089] Step 302: For each target consumer, the traffic characteristic data received by the target consumer is written into the channel of the target consumer corresponding to each traffic characteristic data type dimension through the target consumer.

[0090] In an embodiment of the present application, for each target consumer, the distributed data processing module 130 writes the traffic characteristic data received by the target consumer into the channel of the target consumer corresponding to each traffic characteristic data type dimension through the target consumer. Specifically, for each target consumer, the target consumer writes the traffic characteristic data received by the target consumer into the channel of the target consumer corresponding to each traffic characteristic data type dimension. Each target consumer corresponds to a group of channels, and each traffic characteristic data type dimension corresponds to a channel in the group of channels. The number of channels corresponding to each target consumer may be the same as the number of traffic characteristic data type dimensions.

[0091] Step 303, for each traffic feature data type dimension, through the coroutine of the target consumer corresponding to the traffic feature data type dimension, according to the preset time dimension, count the traffic feature data in the channel of the target consumer corresponding to the traffic feature data type dimension.

[0092] In the embodiment of the present application, each traffic feature data type dimension corresponds to a coroutine of the target consumer. The number of coroutines corresponding to each target consumer may be the same as the number of traffic feature data type dimensions. The channels and coroutines of the same target consumer may correspond one to one.

[0093] Step 304: write the statistically analyzed traffic feature data into the write channel of the target consumer corresponding to the traffic feature data type dimension.

[0094] In an embodiment of the present application, each flow feature data type dimension corresponds to a write channel (writer channel) of a target consumer. The number of write channels (writer channels) corresponding to each target consumer can be the same as the number of flow feature data type dimensions. The channels and write channels (writer channels) of the same target consumer can correspond one to one.

[0095] Step 305: If the preset database writing condition is met, the statistical traffic characteristic data in the writing channel is written into the target database.

[0096] In the embodiment of the present application, the database writing condition may be that the amount of data to be stored in the writing channel is greater than a preset threshold of the amount of data to be stored, or that a preset storage interval is reached. The threshold of the amount of data to be stored may be 100. The storage interval may be 1 minute.

[0097] In one example, if the amount of data to be stored in the write channel is greater than a preset threshold of the amount of data to be stored, or reaches a preset storage interval, the distributed data processing module 130 writes the statistical traffic characteristic data in the write channel into the target database.

[0098] In one embodiment, the schematic diagram of the distributed data processing module 130 using multiple consumers and multiple goroutines to process traffic feature data is shown in Figure 4. The target consumers include consumer 1 and consumer 2. Consumer 1 and consumer 2 receive traffic feature data for access logs and attack logs from different streams. Consumer 1 writes the traffic feature data of interest such as tenants, domain names, source IPs, etc. into different channels (referred to as chan), and different goroutines that process different traffic feature data types and dimensions process the data in the corresponding channels respectively. After processing and statistics, they are written into different writer channels (referred to as writers) to be stored. When the data to be stored in the writer channel is greater than the set threshold, which is one hundred by default, or reaches the default time, which is one minute by default, the data in the writer channel is written into the target database (DB). The specific process of consumer 2 processing traffic feature data is the same as the specific process of consumer 1 processing traffic feature data.

[0099] In the above traffic monitoring method, multiple target consumers are enabled according to the current business volume of each target cloud native WAF node; for each target consumer, the traffic feature data received by the target consumer is written into the channel of the target consumer corresponding to each traffic feature data type dimension through the target consumer; for each traffic feature data type dimension, the traffic feature data in the channel of the target consumer corresponding to the traffic feature data type dimension is counted according to the preset time dimension through the coroutine of the target consumer corresponding to the traffic feature data type dimension; the traffic feature data after statistics is written into the write channel of the target consumer corresponding to the traffic feature data type dimension; if the preset database write condition is met, the traffic feature data after statistics written into the channel is written into the target database. In this way, the method adopts a multi-consumer multi-coroutine architecture to process the data of the data aggregation center in parallel. The data in a stream can be processed by multiple different consumers, and different independent coroutines are used for each monitoring focus dimension. It is more suitable for the distributed architecture of the cloud native WAF system and can improve the efficiency of data processing. Moreover, the distributed data processing module 130, as a data processing center, supports multiple consumers to process collected data at the same time. As the WAF business volume increases or decreases, the number of consumers can be gradually increased or decreased, and the support for WAF business monitoring can be smoothly expanded or reduced, thus supporting more convenient business expansion. The method adopts a multi-consumer multi-coroutine architecture, which can also more conveniently realize the expansion of new monitoring dimensions and further enhance scalability.

[0100] In one embodiment, the method further includes the following steps: checking the amount of unconsumed data in the flow at every preset inspection time period; if the amount of unconsumed data is greater than the preset first threshold value of the amount of unconsumed data, determining that the flow has a message backlog and printing an error log; if the amount of unconsumed data is greater than the preset second threshold value of the amount of unconsumed data, outputting an alarm message containing the amount of unconsumed data; and determining the indication information of the increase in the number of consumers based on the alarm message. The inspection time period may be 5s. The indication information of the increase in the number of consumers is used to indicate whether to increase the number of consumers. For example, the traffic monitoring system 100 links the alarm system and outputs an alarm message containing the amount of unconsumed data by sending an enterprise WeChat.

[0101] In an exemplary embodiment, as shown in FIG5 , the method further includes the following steps:

[0102] Step 501, when the preset first time node is reached, the target historical data of the cloud native WAF system is counted and the target historical data is recorded in a temporary file.

[0103] The target historical data includes the historical request volume, the historical attack volume and the historical quantity of each status code. A first time period is spaced between two adjacent first time nodes.

[0104] In the embodiment of the present application, the first time node is a plurality of preset time points. For example, the first time node may be a time point every minute, and accordingly, the first time period is 1 minute. The temporary file is used to cache and temporarily record data.

[0105] Step 502: When the preset second time node is reached, the target historical data between the previous second time node and the current second time node stored in the temporary file is written into the target database.

[0106] There is a second time period between two adjacent second time nodes, and the duration of the second time period is an integer multiple of the duration of the first time period.

[0107] In an embodiment of the present application, the second time node is a plurality of preset time points. In one example, the first time node is a time point every minute, and the first time period is 1 minute; the second time node is a time point every ten minutes, and the second time period is 10 minutes. For example, when the second time node is a time point every ten minutes, the second time node can be 0:00, 0:10, 0:20, ..., 23:50, or 0:05, 0:15, 0:25, ..., 23:55.

[0108] In the above-mentioned traffic monitoring method, when the preset first time node is reached, the target historical data of the cloud native WAF system is counted, and the target historical data is recorded in a temporary file; when the preset second time node is reached, the target historical data between the previous second time node and the current second time node stored in the temporary file is written into the target database. In this way, every second time period, the number of accesses, attacks and response status codes in the current second time period is recorded, which can show the growth of the cloud native WAF business and the current request volume and take into account the database performance. In addition, the data statistics are executed once every first time period, and the statistical results are recorded in a temporary file after execution. After multiple statistics, the statistical data are stored in the database, which can avoid occupying too many database resources and balance the database processing capacity. Moreover, when summarizing and recording historical data, multiple statistics and multiple table records are used to record historical access data, which can ensure the correct recording of historical data. In addition, historical request data can reflect the changes in the business volume, requests, attacks and status codes of WAF products over a long period of time, which can improve the comprehensiveness of traffic monitoring.

[0109] In an exemplary embodiment, as shown in FIG6 , the specific process of writing the target historical data between the last second time node and the current second time node stored in the temporary file into the target database includes the following steps:

[0110] Step 601, if there is a newly added status code when writing to the target database, modify the database table field corresponding to the status code according to the range of the status code and the newly added status code.

[0111] In the embodiment of the present application, the newly added status code is a status code that has never appeared in the target database before being written. The status code ranges from 100 to 599.

[0112] Step 602: Based on the modified database table fields, the historical number of each status code between the last second time node and the current second time node stored in the temporary file is split into a preset number of status code data tables.

[0113] In the embodiment of the present application, the preset number may be 5. The status code data table is also called a dictionary. The status code data table after splitting may be five data tables of 100-199, 200-299, 300-399, 400-499 and 500-599.

[0114] Step 603, write each status code data table and the total request data table into the target database.

[0115] In an embodiment of the present application, the distributed data processing module 130 writes each status code data table and the total request data table as the same transaction to the target database. Specifically, at the beginning of the transaction, the distributed data processing module 130 dynamically constructs a data insertion command according to the column name and the statistical value, and executes the write code until all status code data tables are processed. Then, the distributed data processing module 130 commits the transaction and determines whether there is an error in the transaction execution. If there is an error, the distributed data processing module 130 performs a rollback process and issues an alarm.

[0116] In the above-mentioned traffic monitoring method, if there is a newly added status code when writing to the target database, the database table field corresponding to the status code is modified according to the range of the status code and the newly added status code; based on the modified database table field, the historical number of each status code between the last second time node and the current second time node stored in the temporary file is split into a preset number of status code data tables; each status code data table and the total request data table are written into the target database. In this way, considering the real-time addition of status codes, the table structure is dynamically modified, and considering the large amount of status code data, the status code data table is split, which can further ensure the correct recording of historical data.

[0117] In an exemplary embodiment, as shown in FIG7 , the method further includes the following steps:

[0118] Step 701: If the current time and the last recording time of the temporary file are in different first time periods and in the same second time period, the time period between the current time and the last recording time is used as the first missing time period.

[0119] In an embodiment of the present application, when the first time node is a time point every minute and the second time node is 0:00, 0:10, 0:20, ..., 23:50, if the current time is 0:01 and the last recording time of the temporary file is 0:09, then the current time and the last recording time of the temporary file are in different first time periods and in the same second time period. If the current time is 0:09 and the last recording time of the temporary file is 0:11, then the current time and the last recording time of the temporary file are in different first time periods and in different second time periods. If the current time is 0:01:12 and the last recording time of the temporary file is 0:01:34, then the current time and the last recording time of the temporary file are in the same first time period and in the same second time period.

[0120] Step 702 , count the target historical data of the cloud native WAF system in the first missing time period, and record the target historical data in the first missing time period into a temporary file.

[0121] In the embodiment of the present application, the distributed data processing module 130 counts the target historical data of the cloud native WAF system in the first missing time period. Then, the distributed data processing module 130 records the target historical data in the first missing time period to a temporary file.

[0122] In the above-mentioned traffic monitoring method, in the scenario where the distributed data processing module 130 is shut down and then restarted, if the current time and the last recording time of the temporary file are in different first time periods and in the same second time period, it is only necessary to restore the cached data from the temporary file. While reducing the recovery process, as much target historical data as possible can be statistically recorded, which can improve the efficiency and comprehensiveness of traffic monitoring.

[0123] In an exemplary embodiment, as shown in FIG8 , the method further includes the following steps:

[0124] Step 801: If the current time and the last recording time of the temporary file are in different second time periods, the earliest retention time of the target historical data retained in the target database is obtained.

[0125] In an embodiment of the present application, the target database can clean up the stored data, giving priority to cleaning up data with an early or long storage time. The earliest retention time is the earliest time corresponding to the target historical data retained in the target database. For example, the target database retains target historical data from 8:00 to 9:00, and the earliest retention time is 8:00, and the target historical data before 8:00 can be the cleaned data.

[0126] Step 802: The latest time between the last recording time and the earliest retention time is used as the starting time for statistics.

[0127] Step 803, taking the time period between the start statistical time and the current time as the second missing time period, starting from the start statistical time, statistically processing the target historical data of the cloud native WAF system within the second missing time period according to the first time node and the second time node.

[0128] In the embodiment of the present application, it can be understood that the specific process of the distributed data processing module 130 performing statistical processing on the target historical data of the cloud native WAF system within the second missing time period according to the first time node and the second time node starting from the start statistical time is similar to steps 501-502.

[0129] In one embodiment, if the current time and the last recording time of the temporary file are in the same first time period, no other operations are required. If the distributed data processing module 130 is running for the first time, steps 501-502 are executed.

[0130] In the above-mentioned traffic monitoring method, in the scenario where the distributed data processing module 130 is shut down and then restarted, if the current time and the last recorded time of the temporary file are in a different second time period, the recorded time in the temporary file and the earliest retention time of the target historical data retained in the target database are compared, and statistics are recorded from the latest time. Different data recovery logics are executed for the first run, restart within the first time period, restart within the second time period, and long-term restart, respectively. While reducing the recovery process, as much access attack history data as possible can be statistically recorded, which can improve the efficiency and comprehensiveness of traffic monitoring.

[0131] In one embodiment, the traffic monitoring system 100 scans the amount of data stored in the target database at every preset scanning time period. If the amount of stored data exceeds the preset number of configuration lines, the traffic monitoring system 100 exports the earliest retained data exceeding the preset number of configuration lines and saves it in the export file, and deletes the exported data in the target database. If the retention time of the existing data exceeds the preset retention time threshold, the traffic monitoring system 100 exports the data and saves it in the export file, and deletes the exported data in the target database. When the preset integration time is reached, the traffic monitoring system 100 integrates and packages the exported export files and places them in a unified storage location. Among them, the scanning time period can be 1 minute, that is, the above scanning operation can be performed once every minute. The number of configuration lines can be in the millions, for example, it can be one million. The retention time threshold can be in the hourly level, for example, it can be 2 hours. The export file can be a csv file. Two adjacent integration times can be separated by 1 hour, for example, the integration time can be every hour. The unified storage location can be a disk. The integration and packaging process can be aggregate compression. The traffic monitoring system 100 may also include a cleaning module, and the specific process in this embodiment may also be executed by the cleaning module. In this way, the number of rows of database table records is limited to millions, and only specific processing records within one minute are retained in extreme cases. For data exceeding the configured number, the default is one million data, which will be cleaned up regularly, which can ensure the response speed of the system, take into account performance and availability, and provide more real-time cloud native WAF traffic monitoring.

[0132] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0133] Based on the same inventive concept, the embodiment of the present application also provides a flow monitoring device for implementing the flow monitoring method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more flow monitoring device embodiments provided below can refer to the limitations on the flow monitoring method above, and will not be repeated here.

[0134] In an exemplary embodiment, as shown in FIG9 , a flow monitoring device 900 is provided, including: a first acquisition module 910, an aggregation module 920, a first statistics module 930 and a construction module 940, wherein:

[0135] The first acquisition module 910 is used to respectively acquire the traffic characteristic data of each target cloud native web application firewall WAF node;

[0136] Aggregation module 920, used to aggregate and store traffic characteristic data of each target cloud native WAF node;

[0137] The first statistical module 930 is used to count the traffic characteristic data of each target cloud native WAF node under each traffic characteristic data type dimension according to a preset time dimension, and store the statistical results in a target database;

[0138] The construction module 940 is used to construct a traffic monitoring view according to the data stored in the target database, the predetermined traffic monitoring dimensions and the preset traffic monitoring view construction rules; the traffic monitoring view is used to monitor the cloud native WAF system.

[0139] Optionally, the first statistics module 930 is specifically configured to:

[0140] According to the current business volume of each target cloud native WAF node, multiple target consumers are enabled;

[0141] For each target consumer, the traffic characteristic data received by the target consumer is written into the channel of the target consumer corresponding to each traffic characteristic data type dimension through the target consumer;

[0142] For each traffic feature data type dimension, through the coroutine of the target consumer corresponding to the traffic feature data type dimension, according to the preset time dimension, statistics are collected on the traffic feature data in the channel of the target consumer corresponding to the traffic feature data type dimension;

[0143] Writing the traffic feature data after statistics into the write channel of the target consumer corresponding to the traffic feature data type dimension;

[0144] If the preset database writing condition is met, the statistically analyzed traffic characteristic data in the writing channel is written into the target database.

[0145] Optionally, the device 900 further includes:

[0146] A second statistical module is used to count the target historical data of the cloud native WAF system when a preset first time node is reached, and record the target historical data to a temporary file; the target historical data includes the historical request volume, the historical attack volume and the historical number of each status code; and a first time period is separated between two adjacent first time nodes;

[0147] A writing module is used to write the target historical data between the previous second time node and the current second time node stored in the temporary file into the target database when a preset second time node is reached; a second time period is separated from two adjacent second time nodes; and the length of the second time period is an integer multiple of the length of the first time period.

[0148] Optionally, the writing module is specifically used for:

[0149] If a new status code is added when writing to the target database, modify the database table field corresponding to the status code according to the range of the status code and the new status code;

[0150] Based on the modified database table field, split the historical number of each status code between the previous second time node and the current second time node stored in the temporary file into a preset number of status code data tables;

[0151] Write each of the status code data tables and the total request data table into the target database.

[0152] Optionally, the device 900 further includes:

[0153] A first determining module, configured to, if the current time and the last recording time of the temporary file are in different first time periods and in the same second time period, use the time period between the current time and the last recording time as a first missing time period;

[0154] The third statistical module is used to count the target historical data of the cloud native WAF system in the first missing time period, and record the target historical data in the first missing time period to the temporary file.

[0155] Optionally, the device 900 further includes:

[0156] A second acquisition module, configured to acquire the earliest retention time of the target historical data retained in the target database if the current time and the last recording time of the temporary file are in different second time periods;

[0157] A second determining module is used to use the latest time between the last recording time and the earliest retention time as the starting statistical time;

[0158] The fourth statistical module is used to take the time period between the start statistical time and the current time as the second missing time period, and starting from the start statistical time, perform statistical processing on the target historical data of the cloud native WAF system within the second missing time period according to the first time node and the second time node.

[0159] Each module in the above-mentioned flow monitoring device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each of the above modules.

[0160] In an exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be shown in FIG10. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, the memory, and the input / output interface are connected via a system bus, and the communication interface, the display unit, and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner may be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a flow monitoring method is implemented. The display unit of the computer device is used to form a visually visible picture, which may be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0161] Those skilled in the art will understand that the structure shown in FIG. 10 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0162] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0163] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0164] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0165] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0166] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0167] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0168] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A flow monitoring method, characterized in that: The method comprises: For each target cloud native web application firewall WAF node, obtain the traffic characteristic data of the target cloud native WAF node respectively; Aggregate and store traffic characteristic data of each target cloud native WAF node; According to the preset time dimension, the traffic characteristic data of each target cloud native WAF node under each traffic characteristic data type dimension is counted, and the statistical results are stored in the target database; A traffic monitoring view is constructed based on the data stored in the target database, predetermined traffic monitoring dimensions and preset traffic monitoring view construction rules; the traffic monitoring view is used to monitor the cloud native WAF system.

2. The method according to claim 1, characterized in that The method of counting the traffic characteristic data of each target cloud native WAF node under each traffic characteristic data type dimension according to the preset time dimension, and storing the statistical results in the target database, includes: According to the current business volume of each target cloud native WAF node, multiple target consumers are enabled; For each target consumer, the traffic characteristic data received by the target consumer is written into the channel of the target consumer corresponding to each traffic characteristic data type dimension through the target consumer; For each traffic feature data type dimension, through the coroutine of the target consumer corresponding to the traffic feature data type dimension, according to the preset time dimension, statistics are collected on the traffic feature data in the channel of the target consumer corresponding to the traffic feature data type dimension; Writing the traffic feature data after statistics into the write channel of the target consumer corresponding to the traffic feature data type dimension; If the preset database writing condition is met, the statistically analyzed traffic characteristic data in the writing channel is written into the target database.

3. The method according to claim 1, characterized in that The method further comprises: When a preset first time node is reached, the target historical data of the cloud native WAF system is counted and recorded in a temporary file; the target historical data includes the historical request volume, the historical attack volume and the historical quantity of each status code; and a first time period is separated between two adjacent first time nodes; When the preset second time node is reached, the target historical data between the previous second time node and the current second time node stored in the temporary file is written into the target database; a second time period is separated from two adjacent second time nodes; and the length of the second time period is an integer multiple of the length of the first time period.

4. The method according to claim 3, characterized in that The step of writing the target historical data between the last second time node and the current second time node stored in the temporary file into the target database includes: If a new status code is added when writing to the target database, modify the database table field corresponding to the status code according to the range of the status code and the new status code; Based on the modified database table field, split the historical number of each status code between the previous second time node and the current second time node stored in the temporary file into a preset number of status code data tables; Write each of the status code data tables and the total request data table into the target database.

5. The method according to claim 3, characterized in that: The method further comprises: If the current time and the last recording time of the temporary file are in different first time periods and in the same second time period, the time period between the current time and the last recording time is used as the first missing time period; The target historical data of the cloud native WAF system in the first missing time period is counted, and the target historical data in the first missing time period is recorded in the temporary file.

6. The method according to claim 3, characterized in that The method further comprises: If the current time and the last recording time of the temporary file are in different second time periods, obtaining the earliest retention time of the target historical data retained in the target database; The latest time between the last recording time and the earliest retention time is used as the starting time for statistics; The time period between the statistical start time and the current time is taken as the second missing time period. Starting from the statistical start time, according to the first time node and the second time node, statistical processing is performed on the target historical data of the cloud native WAF system within the second missing time period.

7. A flow monitoring system, characterized in that: The system comprises: At least one distributed data collection module, used for collecting traffic characteristic data of each target cloud native network application firewall WAF node respectively; each of the target cloud native WAF nodes includes one distributed data collection module; A data aggregation center, used to aggregate and store traffic characteristic data of each target cloud native WAF node; A distributed data processing module is used to count the traffic characteristic data of each target cloud native WAF node under each traffic characteristic data type dimension according to a preset time dimension, and store the statistical results in a target database; The storage service module is used to build a traffic monitoring view based on the data stored in the target database, predetermined traffic monitoring dimensions and preset traffic monitoring view building rules; the traffic monitoring view is used to monitor the cloud native WAF system.

8. A flow monitoring device, characterized in that: The device comprises: The first acquisition module is used to respectively acquire the traffic characteristic data of each target cloud native network application firewall WAF node; An aggregation module, used to aggregate and store traffic characteristic data of each target cloud native WAF node; A first statistical module is used to count the traffic characteristic data of each target cloud native WAF node under each traffic characteristic data type dimension according to a preset time dimension, and store the statistical results in a target database; A construction module is used to construct a traffic monitoring view based on the data stored in the target database, predetermined traffic monitoring dimensions and preset traffic monitoring view construction rules; the traffic monitoring view is used to monitor the cloud native WAF system.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images