A flow data collection method, device, equipment and storage medium

Abstract

The present invention discloses a flow data collection method, device, equipment and storage medium, including: receiving multiple data collection strategies, transmitting each data collection strategy to the corresponding hierarchical processing module according to the communication layer corresponding to each data collection strategy; the data collection strategy is pre-set according to business needs; executing multiple data collection strategies through each hierarchical processing module, and monitoring the resources occupied during the operation of the DPI system in real time to obtain resource occupation results; if the resource occupation result exceeds the preset threshold, then determining the target collection strategy to be executed in the multiple data collection strategies in turn according to the priority corresponding to each data collection strategy through each hierarchical processing module, and executing the target collection strategy in turn. The technical solution of the embodiment of the present invention can realize the on-demand collection of flow data in the mobile Internet, and meet the balance between business needs and resource occupation.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of computer technology, and in particular to a flow data collection method, device, equipment and storage medium. Background Art

[0002] With the development of mobile Internet, the analysis of traffic data of mobile devices has become an important task in the mobile Internet industry. Through the analysis of traffic data of mobile devices, we can understand the frequency, time period, type ratio, etc. of users' visits to websites and software, which has a high reference value for the improvement decision and future direction of mobile Internet companies.

[0003] At present, as an effective data collection tool, Deep Packet Inspection (DPI) system is widely used in the field of mobile Internet traffic data collection. Existing DPI systems usually collect mobile Internet traffic data by full collection or sampling collection.

[0004] However, the full data collection method is costly, has significant delays in implementation and expansion, and does not support flexible business changes. Although the sampling collection method can reduce the amount of data collected, the data is random and cannot fundamentally meet the goal of on-demand data collection, and it is difficult to fully meet the business diversity needs of the business analysis department. Summary of the invention

[0005] The embodiments of the present invention provide a flow data collection method, device, equipment and storage medium, which can realize on-demand collection of flow data in the mobile Internet and meet the balance between business needs and resource occupation.

[0006] In a first aspect, an embodiment of the present invention provides a method for collecting traffic data, which is applied in a deep packet inspection (DPI) system, and includes:

[0007] Receiving multiple data collection strategies, and transmitting each of the data collection strategies to a corresponding hierarchical processing module according to a communication layer corresponding to each of the data collection strategies; the data collection strategies are pre-set according to business requirements;

[0008] Execute multiple data collection strategies through each of the hierarchical processing modules, and monitor the resources occupied during the operation of the DPI system in real time to obtain resource occupancy results;

[0009] If the resource occupancy result exceeds a preset threshold, each of the hierarchical processing modules determines the target acquisition strategy to be executed in turn from a plurality of data acquisition strategies according to the priority corresponding to each of the data acquisition strategies, and executes the target acquisition strategy in turn.

[0010] In a second aspect, an embodiment of the present invention further provides a flow data collection device, which is applied to a data packet deep inspection DPI system, and the device includes:

[0011] A strategy receiving module, used for receiving a plurality of data collection strategies, and transmitting each of the data collection strategies to a corresponding hierarchical processing module according to the communication layer corresponding to each of the data collection strategies; the data collection strategies are pre-set according to business requirements;

[0012] A resource monitoring module, which is used to execute multiple data collection strategies through each of the hierarchical processing modules, and monitor the resources occupied during the operation of the DPI system in real time to obtain resource occupancy results;

[0013] The strategy execution module is used to determine the target collection strategy to be executed in sequence from among multiple data collection strategies through each of the hierarchical processing modules according to the priority corresponding to each of the data collection strategies if the resource occupancy result exceeds a preset threshold, and execute the target collection strategy in sequence.

[0014] In a third aspect, an embodiment of the present invention further provides a computer device, the device comprising:

[0015] one or more processors;

[0016] A storage device for storing one or more programs;

[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement a flow data collection method provided by any embodiment of the present invention.

[0018] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, a flow data collection method provided by any embodiment of the present invention is implemented.

[0019] The technical solution of the embodiment of the present invention receives multiple data collection strategies, transmits each data collection strategy to the corresponding hierarchical processing module according to the communication layer corresponding to each data collection strategy, executes multiple data collection strategies through each hierarchical processing module, and monitors the resources occupied during the operation of the DPI system in real time to obtain resource occupancy results. If the resource occupancy result exceeds a preset threshold, each hierarchical processing module determines the target collection strategy to be executed in the multiple data collection strategies according to the priority corresponding to each data collection strategy, and executes the target collection strategy in sequence. This technical means can realize on-demand collection of traffic data in the mobile Internet and meet the balance between business needs and resource occupancy. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a flow chart of a flow data collection method in Embodiment 1 of the present invention;

[0021] Figure 2 is a flow chart of a flow data collection method in Embodiment 2 of the present invention;

[0022] Figure 3a is a flow chart of a flow data collection method in Embodiment 3 of the present invention;

[0023] Figure 3b It is a structural schematic diagram of a DPI system in Embodiment 3 of the present invention;

[0024] Figure 4 It is a structural diagram of a flow data acquisition device in Embodiment 4 of the present invention;

[0025] Figure 5 It is a structural diagram of a computer device in Embodiment 5 of the present invention. DETAILED DESCRIPTION

[0026] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0027] Embodiment 1

[0028] Figure 1 This is a flow chart of a flow data collection method provided in Embodiment 1 of the present invention. This embodiment is applicable to the case where a DPI system collects flow data in a mobile Internet. The method can be executed by a flow data collection device, which can be implemented by software and / or hardware and can generally be integrated in a terminal or server with a data processing function. Specifically, the method includes the following steps:

[0029] Step 110 , receiving a plurality of data collection strategies, and transmitting each of the data collection strategies to a corresponding hierarchical processing module according to a communication layer corresponding to each of the data collection strategies.

[0030] In this embodiment, the data collection strategy is pre-set according to business requirements. Specifically, the data collection strategy can be pre-set manually according to specific business requirements (eg, collecting flow data of a target interface in a target business).

[0031] In this step, after receiving multiple data collection strategies, the communication layer corresponding to each data collection strategy can be determined according to the source of the data to be collected specified in each data collection strategy. Specifically, assuming that the data collection strategy is "collecting the original message data of the domain name resolution (DNS) with the request domain name xxxx", it can be determined that the communication layer corresponding to the data collection strategy is the network layer.

[0032] In this embodiment, hierarchical processing modules corresponding to each communication layer are pre-deployed in the DPI system. After the communication layer corresponding to the data collection strategy is determined in the above manner, the data collection strategy can be transmitted to the corresponding hierarchical processing module. Specifically, assuming that the communication layer corresponding to a certain data collection strategy is the network layer, the data collection strategy can be transmitted to the network layer processing module in the DPI system.

[0033] Step 120: Execute multiple data collection strategies through each of the hierarchical processing modules, and monitor the resources occupied during the operation of the DPI system in real time to obtain resource occupation results.

[0034] In this embodiment, after receiving the matching data collection strategy, each of the hierarchical processing modules can execute multiple data collection strategies in parallel, and during the execution process, monitor the resources occupied during the operation of the DPI system in real time to obtain resource occupancy results.

[0035] In a specific embodiment, each of the hierarchical processing modules may monitor the computing power, memory, bandwidth and other resources of the central processing unit (CPU) occupied during the operation of the DPI system to obtain resource occupancy results.

[0036] Step 130: If the resource occupancy result exceeds a preset threshold, each of the hierarchical processing modules determines the target acquisition strategy to be executed in turn from multiple data acquisition strategies according to the priority corresponding to each data acquisition strategy, and executes the target acquisition strategy in turn.

[0037] In this step, if the resource occupancy result exceeds the preset threshold, each of the hierarchical processing modules can determine the target collection strategy to be executed among multiple data collection strategies according to the priority corresponding to each data collection strategy through the optimal resource occupancy reduction algorithm, and stop the execution of other data collection strategies to meet the balance between business needs and resource occupancy.

[0038] In a specific embodiment, the resource occupancy reduction algorithm may include at least one of the following: First-Come First-Service (FCFS) scheduling algorithm, Shortest Job First (SJF) scheduling algorithm, priority scheduling algorithm, high response ratio priority scheduling algorithm, time slice round-robin scheduling algorithm and multi-level feedback queue scheduling algorithm, etc.

[0039] In this embodiment, by receiving multiple data collection strategies formulated based on business needs, on-demand collection of traffic data can be achieved; by deploying hierarchical processing modules corresponding to each communication layer in the DPI system, the hierarchical nature of the data collection strategy can be achieved, making the interface of the traffic data to be collected richer and meeting the business diversity needs of the business department; secondly, according to the priority corresponding to each of the data collection strategies, the target collection strategy to be executed is determined in turn among multiple data collection strategies through the optimal resource occupancy reduction algorithm, which can maximize the balance between business needs and resource occupancy.

[0040] The technical solution of the embodiment of the present invention receives multiple data collection strategies, transmits each data collection strategy to the corresponding hierarchical processing module according to the communication layer corresponding to each data collection strategy, executes multiple data collection strategies through each hierarchical processing module, and monitors the resources occupied during the operation of the DPI system in real time to obtain resource occupancy results. If the resource occupancy result exceeds a preset threshold, each hierarchical processing module determines the target collection strategy to be executed in the multiple data collection strategies according to the priority corresponding to each data collection strategy, and executes the target collection strategy in sequence. This technical means can realize on-demand collection of traffic data in the mobile Internet and meet the balance between business needs and resource occupancy.

[0041] Embodiment 2

[0042] This embodiment is a further refinement of the above embodiment. The explanation of terms that are the same or corresponding to those in the above embodiment will not be repeated in this embodiment. Figure 2 This is a flow chart of a flow data collection method provided in the second embodiment. In this embodiment, the technical solution of this embodiment can be combined with one or more methods in the solutions of the above embodiments, such as Figure 2 As shown, the method provided in this embodiment may also include:

[0043] Step 210: Receive multiple data collection strategies.

[0044] Step 220: Convert each of the data collection strategies into data collection instructions suitable for execution by the DPI system.

[0045] In this step, each data collection strategy can be converted into a data collection instruction suitable for execution by the DPI system in a unified conversion format according to the communication layer corresponding to the data collection strategy, the business type specified in the data collection strategy, the data type of the data to be collected, and the time period.

[0046] In this embodiment, the data collection strategy that needs to be converted usually corresponds to the following scenarios: network layer original message data, transport layer original message data-Internet Protocol (IP) quintuple, transport layer original message data-specific location payload, application layer original message data, and log data under specific conditions.

[0047] Step 230 , judging whether there is a conflict between the data collection instructions according to the correlation between the data collection instructions, if so, executing steps 240 - 260 , if not, executing step 270 .

[0048] In this embodiment, when determining that two data acquisition instructions are associated with each other, it is possible to judge whether there are overlap conflicts, association conflicts, redundancy conflicts, generalization conflicts, and irrelevant exceptions between the data acquisition instructions by comparing the relationship between the data acquisition instructions and the action behaviors of the data acquisition instructions.

[0049] In a specific embodiment, it can be determined whether the paths and coverage of two data acquisition instructions in the strategy tree are consistent. If the path of the current data acquisition instruction is consistent with the path and coverage of the previous data acquisition instruction, then there is a high probability that there is a conflict between the two data acquisition instructions; if the paths of the two data acquisition instructions are inconsistent, then there is no conflict between the two data acquisition instructions.

[0050] In one implementation of this embodiment, the relationship between the data collection instructions includes:

[0051] Exact match relationship: The values ​​of all policy items in the data collection instruction are equal, and there is an exact match between the two data collection instruction items.

[0052] Inclusion match: Data collection instructions A and data collection instructions B are not completely matched, and the value of each policy item of A is a subset of or equal to the value of the corresponding policy item of B, then A and B are inclusion matches.

[0053] Completely unrelated: All policy items in data collection instruction A are not equal to the values ​​of corresponding policy items in data collection instruction B, and there is no subset or superset relationship, so A and B are completely unrelated.

[0054] Partial match: If the value of at least one policy item in data collection instruction A is equal to the value of the corresponding policy item in data collection instruction B, or there is a subset or super relationship, and the value of at least one policy item is not equal to the value of the corresponding policy item in data collection instruction B, and there is no subset or super relationship, then A and B partially match.

[0055] Association relationship: If there are some policy items in data collection instruction A whose values ​​are subsets or equal to the corresponding policy items of data collection instruction B, and the values ​​of other policy items of A are supersets of the values ​​of the corresponding policy items of B, then A and B are associated.

[0056] In a specific embodiment, assuming that data collection instruction A is to collect DNS original message data (transport layer is UDP protocol, port number 53), and data collection instruction B is to collect DNS original message data with request domain name xxxx.com, it can be determined that the two data collection instructions have a strong correlation.

[0057] Step 240: Obtain the conflicting first data collection instruction and the second data collection instruction.

[0058] Step 250: Determine an invalid data collection instruction according to the priorities respectively corresponding to the first data collection instruction and the second data collection instruction, and uninstall the invalid data collection instruction.

[0059] In a specific embodiment, assuming that the priority of the first data acquisition instruction is higher than the priority of the second data acquisition instruction, the first data acquisition instruction can be executed, and the second data acquisition instruction can be uninstalled as an invalid acquisition instruction.

[0060] In one implementation of this embodiment, before determining the invalid data collection instruction according to the priorities respectively corresponding to the first data collection instruction and the second data collection instruction, the method further includes:

[0061] Step 241: if the first data collection instruction and the second data collection instruction are original message collection instructions, determining the priorities of the first data collection instruction and the second data collection instruction respectively according to the communication layers respectively corresponding to the first data collection instruction and the second data collection instruction;

[0062] In a specific embodiment, it is assumed that the first data collection instruction is to collect DNS original message data (the transport layer is UDP protocol, port number 53), and the second data collection instruction is to collect DNS original message data with the request domain name being xxxx.com. Because the second data collection instruction is within the scope of the first data collection instruction, and the communication layer corresponding to the second data collection instruction is above the first data collection instruction, the first data collection instruction has a higher priority.

[0063] In another specific embodiment, it is assumed that the first data collection instruction is to collect all the original message data of the Transmission Control Protocol (TCP), and the second data collection instruction is to collect the original message data of the Hypertext Transfer Protocol (HTTP). Because the second data collection instruction is within the scope of the first data collection instruction, and the communication layer corresponding to the second data collection instruction is above the first data collection instruction, the first data collection instruction has a higher priority.

[0064] Step 242: If the first data collection instruction and the second data collection instruction are log collection instructions, determine the priorities of the first data collection instruction and the second data collection instruction respectively according to the data ranges respectively covered by the first data collection instruction and the second data collection instruction.

[0065] In a specific embodiment, assuming that the first data collection instruction is to collect log data of a certain city, and the second data collection instruction is to collect data from N base stations, if these N base stations are all within the city, it can be determined that the first data collection instruction covers a larger data range, that is, the first data collection instruction has a higher priority.

[0066] Step 260: According to the communication layers corresponding to the remaining data collection instructions, the corresponding data collection strategies are transmitted to the matching hierarchical processing modules respectively.

[0067] Step 270: According to the communication layer corresponding to each of the data collection instructions, the corresponding data collection strategies are transmitted to the matching hierarchical processing modules respectively.

[0068] Step 280: Execute multiple data collection strategies through each of the hierarchical processing modules, and monitor the resources occupied during the operation of the DPI system in real time to obtain resource occupation results.

[0069] In one implementation of this embodiment, multiple data collection strategies are executed by each of the hierarchical processing modules, and the resources occupied during the operation of the DPI system are monitored in real time to obtain resource occupation results, including:

[0070] Step 281: Execute multiple data collection strategies through each of the hierarchical processing modules to monitor the hardware device resources occupied during the operation of the DPI system in real time to obtain hardware resource occupation results;

[0071] In this step, the CPU resources occupied during the operation of the DPI system, CPU load status, storage occupancy, data reporting bandwidth resources, and load resources occupied by key modules can be monitored in real time to obtain hardware resource occupancy results.

[0072] Step 282: Obtain threads called by the DPI system in the process of executing multiple data collection strategies, monitor the software resources occupied by all threads, and obtain software resource occupation results.

[0073] In this step, the software resources occupied by all threads can be monitored, such as the calling frequency of the threads, the data volume of the threads, the response time of the threads, etc., to obtain the software resource occupation result.

[0074] In this embodiment, the above resource usage results can be displayed in the form of a dashboard to control the overall performance of the DPI system from a macro perspective. The data volume of each data collection strategy and its impact on system performance can also be intuitively displayed in the form of a multi-dimensional report.

[0075] In a specific embodiment, the business data managed by the data collection strategy can be mined and analyzed to predict the resources occupied by the traffic data during the collection process, and the prediction result can be combined with the above resource occupancy result to jointly update the target collection strategy to be executed.

[0076] Step 290: If the resource occupancy result exceeds a preset threshold, each of the hierarchical processing modules determines the target collection strategy to be executed in turn from among the multiple data collection strategies according to the priority corresponding to each of the data collection strategies, and executes the target collection strategy in turn.

[0077] The technical solution of the embodiment of the present invention receives multiple data collection strategies, converts each data collection strategy into a data collection instruction, and determines whether there is a conflict between the data collection instructions according to the correlation between the data collection instructions. If so, the first data collection instruction and the second data collection instruction in which the conflict occurs are obtained, and the invalid data collection instruction is determined according to the priorities corresponding to the first data collection instruction and the second data collection instruction, and the invalid data collection instruction is uninstalled. According to the communication layer corresponding to the remaining data collection instructions, the corresponding data collection strategies are respectively transmitted to the matching hierarchical processing module; if not, according to the communication layer corresponding to the data collection instruction, the corresponding data collection strategies are respectively transmitted to the matching hierarchical processing module, and finally, the multiple data collection strategies are executed by each hierarchical processing module, and the resources occupied during the operation of the DPI system are monitored in real time to obtain the resource occupation result. If the resource occupation result exceeds the preset threshold, the target collection strategy to be executed in the multiple data collection strategies is determined in turn by each hierarchical processing module according to the priority corresponding to each data collection strategy, and the technical means of executing the target collection strategy in turn can realize on-demand collection of traffic data in the mobile Internet and meet the balance between business needs and resource occupation.

[0078] Embodiment 3

[0079] This embodiment is a further refinement of the above embodiment. The explanation of terms that are the same or corresponding to those in the above embodiment will not be repeated in this embodiment. Figure 3a This is a flow chart of a flow data collection method provided in the third embodiment. In this embodiment, the technical solution of this embodiment can be combined with one or more methods in the solutions of the above embodiments, such as Figure 3a As shown, the method provided in this embodiment may also include:

[0080] Step 310: Receive multiple data collection strategies, and transmit each data collection strategy to a corresponding hierarchical processing module according to the communication layer corresponding to each data collection strategy.

[0081] Step 320: Execute multiple data collection strategies through each of the hierarchical processing modules, and monitor the resources occupied during the operation of the DPI system in real time to obtain resource occupation results.

[0082] Step 330: If the resource occupancy result exceeds a preset threshold, each of the hierarchical processing modules determines the target collection strategy to be executed in turn from among the multiple data collection strategies according to the priority corresponding to each of the data collection strategies, and executes the target collection strategy in turn.

[0083] Step 340: Obtain target traffic data corresponding to each of the plurality of target collection strategies, and determine a data sending method that matches each of the target traffic data according to a data type corresponding to each of the target traffic data.

[0084] In this step, each target traffic data can be divided into the following two data types according to the data transmission frequency and file capacity: non-real-time large batch data and real-time data with small flow. Among them, non-real-time large batch data can be data with large capacity and low frequency, such as business log data or original code stream data; real-time data with small flow can be data with small capacity, high frequency and strong real-time performance, such as real-time alarm data, location change data, and real-time tracking data.

[0085] In this embodiment, if the data type corresponding to the target flow data is non-real-time bulk data, the data sending method matching the target flow data may be: a sending method based on the Web Service protocol and the File Transfer Protocol (FTP);

[0086] If the data type corresponding to the target flow data is real-time data with a small flow rate, the data transmission mode matching the target flow data may be: a transmission mode based on a fast exchange protocol. The fast exchange protocol may be a message queue (MQ) protocol and a socket protocol.

[0087] In one implementation of the present embodiment, after obtaining the target flow data corresponding to multiple target collection strategies respectively, it also includes: verifying each of the target flow data according to at least one data verification standard to obtain a verification result corresponding to each of the target flow data; determining invalid flow data from multiple target flow data according to the verification result corresponding to each of the target flow data, and eliminating the invalid flow data.

[0088] In this step, the data verification standard may include standards such as field accuracy, field completeness, and field reliability. If the target flow data fails to be verified under the data verification standard, the target flow data may be regarded as invalid flow data.

[0089] Step 350: Send each target traffic data to the application end according to the data sending method matching the target traffic data.

[0090] The technical solution of the embodiment of the present invention receives multiple data collection strategies, transmits each data collection strategy to the corresponding hierarchical processing module according to the communication layer corresponding to each data collection strategy, executes multiple data collection strategies through each hierarchical processing module, and monitors the resources occupied during the operation of the DPI system in real time to obtain resource occupancy results. If the resource occupancy result exceeds the preset threshold, the target collection strategy to be executed is determined in turn from the multiple data collection strategies according to the priority corresponding to each data collection strategy through each hierarchical processing module, and the target collection strategy is executed in turn to obtain the target traffic data corresponding to the multiple target collection strategies, and the data type corresponding to each of the target traffic data is determined. The data sending method matching each of the target traffic data is determined, and the technical means of sending each of the target traffic data to the application end according to the data sending method matching each of the target traffic data can realize on-demand collection of traffic data in the mobile Internet and meet the balance between business needs and resource occupancy.

[0091] Based on the above embodiments, Figure 3b is a structural diagram of a DPI system in this embodiment, such as Figure 3b As shown, the DPI system includes a policy delivery module, an elastic policy preprocessing module and a data reporting module. The policy delivery module is used to receive multiple data collection strategies pre-set according to business needs, and transmit each data collection strategy to the corresponding hierarchical processing module; the elastic policy preprocessing module is deployed in each hierarchical processing module to execute each assigned data collection strategy; the data reporting module is used to obtain the traffic data obtained after the execution of each data collection strategy, and send the traffic data to the application end.

[0092] In this embodiment, the policy delivery module includes:

[0093] A strategy receiving unit, used for receiving multiple data collection strategies;

[0094] A strategy translation unit, used to convert each of the data collection strategies into a data collection instruction suitable for execution by the DPI system;

[0095] A policy conflict detection unit, used to determine whether there is a conflict between the data collection instructions according to the correlation between the data collection instructions;

[0096] A strategy execution path selection unit, used to determine the communication layer corresponding to the data collection strategy and the hierarchical processing module matching the data collection strategy, and transmit each of the data collection strategies to the corresponding hierarchical processing module;

[0097] The policy lifecycle management unit is used to manage the entire life cycle of data collection policies, including loading, unloading, and execution duration. Secondly, the policy lifecycle management unit is also used to verify and sort out each data collection policy, optimize hidden policies, redundant policies, empty policies, etc., to avoid business and security risks brought by non-compliant policies, and promptly clean up temporarily enabled or expired data collection policies to avoid policy redundancy.

[0098] The elastic policy preprocessing module includes:

[0099] A strategy receiving unit, used for receiving multiple data collection strategies;

[0100] A policy analysis unit is used to analyze the data collection policy to obtain the data type, business type and specific time period corresponding to the data to be collected;

[0101] The performance monitoring unit is used to monitor the resources (such as the CPU load growth, etc.) occupied during the operation of the DPI system and obtain resource monitoring results;

[0102] The resource evaluation unit is used to predict the resource usage of traffic data during the collection process, and to combine the prediction result with the resource monitoring result to determine whether the predicted resource usage result exceeds a preset threshold;

[0103] A strategy priority determination unit, used to determine the priority matching each data collection strategy;

[0104] A reduction algorithm unit, used to determine a target acquisition strategy to be executed among multiple data acquisition strategies by using an optimal resource occupancy reduction algorithm according to the priorities corresponding to the data acquisition strategies;

[0105] A strategy updating unit, used to update the current data collection strategy to be executed to the above target collection strategy;

[0106] The strategy sending unit is used to send the target acquisition strategy to the corresponding processor to complete the execution of the target acquisition strategy.

[0107] Wherein, the data reporting module includes:

[0108] A data subscription unit is used to intercept corresponding data fields in the traffic data according to actual needs;

[0109] A data transmission unit, used for transmitting the intercepted data field to the filter;

[0110] A data filtering unit, used to filter data fields according to preset rules through filters;

[0111] A data management unit is used to verify each of the flow data according to at least one data verification standard to obtain a verification result corresponding to each of the flow data; according to the verification result corresponding to each of the flow data, determine invalid flow data from the multiple flow data, and eliminate the invalid flow data;

[0112] A data quality unit is used to optimize the traffic data.

[0113] Embodiment 4

[0114] Figure 4 This is a structural diagram of a flow data collection device provided in the fourth embodiment of the present invention. The device is applied in a deep packet inspection (DPI) system and includes: a policy receiving module 410, a resource monitoring module 420 and a policy execution module 430.

[0115] The strategy receiving module 410 is used to receive multiple data collection strategies and transmit each data collection strategy to a corresponding hierarchical processing module according to the communication layer corresponding to each data collection strategy; the data collection strategy is pre-set according to business requirements;

[0116] The resource monitoring module 420 is used to execute multiple data collection strategies through each of the hierarchical processing modules, and monitor the resources occupied during the operation of the DPI system in real time to obtain resource occupation results;

[0117] The strategy execution module 430 is used to determine the target collection strategy to be executed in sequence from among multiple data collection strategies through each of the hierarchical processing modules according to the priority corresponding to each of the data collection strategies if the resource occupancy result exceeds a preset threshold, and execute the target collection strategy in sequence.

[0118] The technical solution of the embodiment of the present invention receives multiple data collection strategies, transmits each data collection strategy to the corresponding hierarchical processing module according to the communication layer corresponding to each data collection strategy, executes multiple data collection strategies through each hierarchical processing module, and monitors the resources occupied during the operation of the DPI system in real time to obtain resource occupancy results. If the resource occupancy result exceeds a preset threshold, each hierarchical processing module determines the target collection strategy to be executed in the multiple data collection strategies according to the priority corresponding to each data collection strategy, and executes the target collection strategy in sequence. This technical means can realize on-demand collection of traffic data in the mobile Internet and meet the balance between business needs and resource occupancy.

[0119] Based on the above embodiments, the policy receiving module 410 may include:

[0120] A strategy conversion unit, used to convert each of the data collection strategies into a data collection instruction suitable for execution by the DPI system;

[0121] A conflict judgment unit, used for judging whether there is a conflict between the data collection instructions according to the correlation between the data collection instructions;

[0122] A strategy transmission unit, used for transmitting the corresponding data collection strategies to the matching hierarchical processing modules according to the communication layers corresponding to the data collection instructions when there is no conflict between the data collection instructions;

[0123] An acquisition instruction acquisition unit, used for acquiring a first data acquisition instruction and a second data acquisition instruction that conflict when there is a conflict between the data acquisition instructions;

[0124] an invalid instruction determining unit, configured to determine an invalid data acquisition instruction according to the priorities respectively corresponding to the first data acquisition instruction and the second data acquisition instruction, and to uninstall the invalid data acquisition instruction;

[0125] The remaining instruction processing unit is used to transmit the corresponding data collection strategies to the matching hierarchical processing modules according to the communication layers corresponding to the remaining data collection instructions;

[0126] A first priority determination unit, configured to determine the priorities of the first data acquisition instruction and the second data acquisition instruction respectively according to the communication layers respectively corresponding to the first data acquisition instruction and the second data acquisition instruction if the first data acquisition instruction and the second data acquisition instruction belong to the original message acquisition instruction;

[0127] The second priority determination unit is used to determine the priorities corresponding to the first data collection instruction and the second data collection instruction respectively according to the data ranges respectively covered by the first data collection instruction and the second data collection instruction if the first data collection instruction and the second data collection instruction are log collection instructions.

[0128] The resource monitoring module 420 may include:

[0129] A hardware resource monitoring unit is used to execute multiple data collection strategies through each of the hierarchical processing modules, monitor the hardware device resources occupied during the operation of the DPI system in real time, and obtain hardware resource occupancy results;

[0130] The software resource monitoring unit is used to obtain the threads called by the DPI system during the execution of multiple data collection strategies, monitor the software resources occupied by all threads, and obtain the software resource occupancy results.

[0131] The policy execution module 430 may include:

[0132] A sending mode determining unit, used for acquiring target flow data corresponding to a plurality of target collection strategies respectively, and determining a data sending mode matching each of the target flow data according to a data type corresponding to each of the target flow data;

[0133] A data sending unit, used for sending each of the target flow data to the application end according to a data sending mode matching the each of the target flow data;

[0134] A data verification unit, configured to verify each of the target flow data according to at least one data verification standard, and obtain a verification result corresponding to each of the target flow data;

[0135] The data elimination unit is used to determine invalid traffic data from a plurality of target traffic data according to the verification results corresponding to each of the target traffic data, and eliminate the invalid traffic data.

[0136] The flow data collection device provided in the embodiment of the present invention can execute the flow data collection method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0137] Embodiment 5

[0138] Figure 5 A schematic diagram of the structure of a computer device provided in Embodiment 5 of the present invention is shown in FIG. Figure 5 As shown, the computer device includes a processor 510, a memory 520, an input device 530, and an output device 540; the number of the processor 510 in the computer device can be one or more. Figure 5 A processor 510 is taken as an example; the processor 510, the memory 520, the input device 530 and the output device 540 in the computer device can be connected via a bus or other means. Figure 5 The example of connecting via a bus is taken. The memory 520, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to a flow data collection method in any embodiment of the present invention (for example, a policy receiving module 410, a resource monitoring module 420 and a policy execution module 430 in a flow data collection device). The processor 510 executes various functional applications and data processing of the computer device by running the software programs, instructions and modules stored in the memory 520, that is, implements the above-mentioned flow data collection method. That is, when the program is executed by the processor, it implements:

[0139] Receiving multiple data collection strategies, and transmitting each of the data collection strategies to a corresponding hierarchical processing module according to a communication layer corresponding to each of the data collection strategies; the data collection strategies are pre-set according to business requirements;

[0140] Execute multiple data collection strategies through each of the hierarchical processing modules, and monitor the resources occupied during the operation of the DPI system in real time to obtain resource occupancy results;

[0141] If the resource occupancy result exceeds a preset threshold, each of the hierarchical processing modules determines the target acquisition strategy to be executed in turn from a plurality of data acquisition strategies according to the priority corresponding to each of the data acquisition strategies, and executes the target acquisition strategy in turn.

[0142] The memory 520 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 520 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 520 may further include a memory remotely arranged relative to the processor 510, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The input device 530 may be used to receive input digital or character information, and to generate key signal input related to user settings and function control of the computer device, and may include a keyboard and a mouse, etc. The output device 540 may include a display device such as a display screen.

[0143] Embodiment 6

[0144] Embodiment 6 of the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the method described in any embodiment of the present invention is implemented. Of course, a computer-readable storage medium provided by an embodiment of the present invention can perform related operations in a flow data collection method provided by any embodiment of the present invention. That is, when the program is executed by a processor, the following is implemented:

[0145] Receiving multiple data collection strategies, and transmitting each of the data collection strategies to a corresponding hierarchical processing module according to a communication layer corresponding to each of the data collection strategies; the data collection strategies are pre-set according to business requirements;

[0146] Execute multiple data collection strategies through each of the hierarchical processing modules, and monitor the resources occupied during the operation of the DPI system in real time to obtain resource occupancy results;

[0147] If the resource occupancy result exceeds a preset threshold, each of the hierarchical processing modules determines the target acquisition strategy to be executed in turn from a plurality of data acquisition strategies according to the priority corresponding to each of the data acquisition strategies, and executes the target acquisition strategy in turn.

[0148] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0149] It is worth noting that in the above-mentioned embodiment of the flow data acquisition device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0150] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A flow data collection method, characterized in that: Applied in deep packet inspection (DPI) systems, including: Receiving multiple data collection strategies, and transmitting each of the data collection strategies to a corresponding hierarchical processing module according to a communication layer corresponding to each of the data collection strategies; the data collection strategies are pre-set according to business requirements; Execute multiple data collection strategies through each of the hierarchical processing modules, and monitor the resources occupied during the operation of the DPI system in real time to obtain resource occupancy results; If the resource occupancy result exceeds a preset threshold, each of the hierarchical processing modules determines the target acquisition strategy to be executed in sequence from among the multiple data acquisition strategies through an optimal resource occupancy reduction algorithm according to the priority corresponding to each of the data acquisition strategies, and executes the target acquisition strategy in sequence; Wherein, according to the communication layer corresponding to each data collection strategy, each data collection strategy is transmitted to the corresponding hierarchical processing module, including: Convert each of the data collection strategies into data collection instructions suitable for execution by the DPI system; According to the correlation between the data collection instructions, determining whether there is a conflict between the data collection instructions; If not, then, according to the communication layer corresponding to each of the data collection instructions, the corresponding data collection strategies are transmitted to the matching hierarchical processing modules respectively; If yes, obtaining the conflicting first data collection instruction and the second data collection instruction; Determine an invalid data collection instruction according to the priorities respectively corresponding to the first data collection instruction and the second data collection instruction, and uninstall the invalid data collection instruction; According to the communication layers corresponding to the remaining data collection instructions, the corresponding data collection strategies are respectively transmitted to the matching hierarchical processing modules.

2. The method according to claim 1, characterized in that Before determining the invalid data collection instruction according to the priorities respectively corresponding to the first data collection instruction and the second data collection instruction, the method further includes: If the first data collection instruction and the second data collection instruction are original message collection instructions, determining the priorities of the first data collection instruction and the second data collection instruction respectively according to the communication layers respectively corresponding to the first data collection instruction and the second data collection instruction; If the first data collection instruction and the second data collection instruction are log collection instructions, the priorities corresponding to the first data collection instruction and the second data collection instruction are determined according to the data ranges respectively covered by the first data collection instruction and the second data collection instruction.

3. The method according to claim 1, characterized in that Multiple data collection strategies are executed through each of the hierarchical processing modules, and the resources occupied during the operation of the DPI system are monitored in real time to obtain resource occupancy results, including: By executing multiple data collection strategies through each of the hierarchical processing modules, the hardware device resources occupied during the operation of the DPI system are monitored in real time to obtain the hardware resource occupancy results; The threads called by the DPI system in the process of executing multiple data collection strategies are obtained, and the software resources occupied by all threads are monitored to obtain the software resource occupancy results.

4. The method according to claim 1, characterized in that: After sequentially determining the target acquisition strategies to be executed from the multiple data acquisition strategies and sequentially executing the target acquisition strategies, the method further includes: Acquire target flow data corresponding to each of the plurality of target collection strategies, and determine a data sending mode matching each of the target flow data according to a data type corresponding to each of the target flow data; Each of the target traffic data is sent to the application end according to a data sending method that matches the each of the target traffic data.

5. The method according to claim 4, characterized in that After obtaining the target traffic data corresponding to the multiple target collection strategies respectively, it also includes: Verifying each of the target flow data according to at least one data verification standard to obtain a verification result corresponding to each of the target flow data; According to the verification results corresponding to each of the target flow data, invalid flow data is determined in the multiple target flow data, and the invalid flow data is eliminated.

6. A flow data collection device, characterized in that: Applied in deep packet inspection (DPI) systems, including: A strategy receiving module, used for receiving a plurality of data collection strategies, and transmitting each of the data collection strategies to a corresponding hierarchical processing module according to the communication layer corresponding to each of the data collection strategies; the data collection strategies are pre-set according to business requirements; A resource monitoring module, which is used to execute multiple data collection strategies through each of the hierarchical processing modules, and monitor the resources occupied during the operation of the DPI system in real time to obtain resource occupancy results; A strategy execution module, configured to determine the target acquisition strategy to be executed in sequence from among the multiple data acquisition strategies by the optimal resource occupancy reduction algorithm according to the priority corresponding to each data acquisition strategy if the resource occupancy result exceeds a preset threshold, and execute the target acquisition strategy in sequence; Wherein, according to the communication layer corresponding to each data collection strategy, each data collection strategy is transmitted to the corresponding hierarchical processing module, including: Convert each of the data collection strategies into data collection instructions suitable for execution by the DPI system; According to the correlation between the data collection instructions, determining whether there is a conflict between the data collection instructions; If not, then, according to the communication layer corresponding to each of the data collection instructions, the corresponding data collection strategies are transmitted to the matching hierarchical processing modules respectively; If yes, obtaining the conflicting first data collection instruction and the second data collection instruction; Determine an invalid data collection instruction according to the priorities respectively corresponding to the first data collection instruction and the second data collection instruction, and uninstall the invalid data collection instruction; According to the communication layers corresponding to the remaining data collection instructions, the corresponding data collection strategies are respectively transmitted to the matching hierarchical processing modules.

7. A computer device comprising: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the flow data collection method as described in any one of claims 1 to 5 when executing the programs.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the flow data collection method as described in any one of claims 1 to 5 is implemented.

Images