Traffic optimization analysis method, device, equipment and medium for collection system

Abstract

The application provides a traffic optimization analysis method, device, equipment and medium for a collection system, the method comprising: collecting traffic through the collection system and uploading the traffic to a traffic processing server in the form of a traffic package; splitting and deduplicating the traffic package according to a set strategy at a fixed time to obtain processed traffic data; setting a trigger to divide the traffic data into risk data and non-risk data and import the data into a database; extracting the risk data and the non-risk data according to a set strategy and performing desensitization to obtain desensitized data; forwarding the risk data to a first setting personnel and forwarding the desensitized risk data to a second setting personnel; and solving the problem of low and unstable analysis and processing efficiency of massive data in a traditional method.

Description

Technical Field

[0001] This invention relates to a method, apparatus, device, and medium for flow optimization analysis of a data acquisition system. Background Technology

[0002] In network management, data acquisition systems are used to capture network data packets and provide the collected information to relevant departments for traffic analysis. However, the traffic data collected by these systems is exported in large, unorganized packets, resulting in poor readability and failing to meet departmental needs. Conventional traffic analysis uses Excel, which requires significant manual intervention, and Excel becomes inadequate for analyzing massive amounts of traffic as the volume increases. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a method, device, equipment and medium for traffic optimization analysis of a data acquisition system, which solves the problems of low efficiency and instability of traditional methods for analyzing and processing massive amounts of data.

[0004] In a first aspect, the present invention provides a method for traffic optimization analysis of a data acquisition system, comprising:

[0005] Step 1: Collect traffic through the acquisition system and upload it to the traffic processing server in the form of traffic packets;

[0006] Step 2: Periodically split and deduplicate the traffic packets according to the set strategy to obtain the processed traffic data;

[0007] Step 3: Set up triggers to categorize traffic data into risky and non-risky data, and import them into the database;

[0008] Step 4: Extract risk data and non-risk data according to the set strategy, and then de-identify them to obtain de-identified data;

[0009] Step 5: Forward the risk data to the first designated person, and then forward the anonymized risk data to the second designated person.

[0010] Furthermore, step 2 is further specified as follows: periodically performing operations on traffic packets using strategies such as removing invalid columns, merging random ports with the same IP address, filtering abnormal data, and labeling data, and obtaining processed traffic data through data cleaning and optimization deduplication.

[0011] Further, step 3 is further specified as follows: setting triggers to divide traffic data into risk data and non-risk data, wherein the risk data is used for abnormal traffic investigation; the risk data and non-risk data are stored in a risk data record table; the triggers include: trigger conditions for a set time period, trigger conditions for a set IP address, trigger conditions for abnormal access, and trigger conditions for a set number of accesses.

[0012] Further, step 5 specifically involves: creating a column named "warn_mes" of type bool in the risk data record table, setting the value of previously sent risk data to 1 and the value of unsent risk data to 0, with a default value of 0; periodically extracting all risk data in the risk data record table where "warn_mes" is 0, and setting the corresponding value of the extracted risk data to 1; for unsent risk data, firstly forming a complete abnormal data file for provision to the first designated personnel; secondly, distributing the de-identified risk data to the corresponding second designated personnel based on the source and target IP of the risk data.

[0013] Secondly, the present invention provides a flow optimization and analysis device for a data acquisition system, comprising:

[0014] The upload module collects traffic through the acquisition system and uploads it to the traffic processing server in the form of traffic packets;

[0015] The import module periodically splits and deduplicates traffic packets according to a set strategy to obtain processed traffic data.

[0016] In the data mining module, triggers are set to categorize traffic data into risky and non-risky data, and then imported into the database.

[0017] The extraction and desensitization module extracts risky and non-risky data according to the set strategy, and then desensitizes the data to obtain desensitized data.

[0018] The risk warning forwarding module forwards risk data to the first designated personnel and then forwards the anonymized risk data to the second designated personnel.

[0019] Furthermore, the import module specifically involves periodically performing operations on traffic packets using strategies such as removing invalid columns, merging random ports with the same IP address, filtering abnormal data, and labeling data. The processed traffic data is obtained through data cleaning and optimization deduplication.

[0020] Furthermore, the mining module is further specified as follows: setting triggers to divide traffic data into risk data and non-risk data, wherein the risk data is used for abnormal traffic investigation; the risk data and non-risk data are stored in a risk data record table; the triggers include: trigger conditions for a set time period, trigger conditions for a set IP address, trigger conditions for abnormal access, and trigger conditions for a set number of accesses.

[0021] Furthermore, the risk warning forwarding module specifically involves: creating a column named "warn_mes" of type bool in the risk data record table, setting the value of previously sent risk data to 1 and the value of unsent risk data to 0, with a default value of 0; periodically extracting all risk data in the risk data record table where "warn_mes" is 0, and setting the corresponding value of the extracted risk data to 1; for unsent risk data, firstly forming a complete abnormal data file for provision to the first designated personnel; secondly, distributing the de-identified risk data to the corresponding second designated personnel based on the source and target IP of the risk data.

[0022] Thirdly, the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in the first aspect.

[0023] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0024] One or more technical solutions provided in the embodiments of the present invention have at least the following technical effects or advantages:

[0025] This application provides a traffic optimization analysis method, apparatus, device, and medium for a data acquisition system. It addresses the problems of low efficiency and instability in traditional methods for analyzing and processing massive amounts of data. The proposed traffic optimization analysis method automatically splits, extracts, deduplicates, and anonymizes large data packets. While supporting massive data processing, it designs customized triggers to automatically alarm for abnormal traffic monitoring, thus solving the problems of difficulty in analyzing large data packets and easy information leakage in native systems. While improving the ability to process massive amounts of data, it also helps to uncover potential abnormal traffic, solving the problem of difficulty in analyzing large data packets in data acquisition systems, freeing up manual labor, improving analysis efficiency, and uncovering the deeper value of network traffic analysis, such as providing reference for firewall configuration.

[0026] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0027] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0028] Figure 1 This is a flowchart of the method in Embodiment 1 of the present invention;

[0029] Figure 2 This is a schematic diagram of the device in Embodiment 2 of the present invention;

[0030] Figure 3 This is a flowchart of Embodiment 5 of the present invention;

[0031] Figure 4 This is a schematic diagram of the GUI interface of the present invention. Detailed Implementation

[0032] Example 1

[0033] like Figure 1 As shown, this embodiment provides a traffic optimization analysis method for a data acquisition system, including:

[0034] A traffic optimization analysis method for a data acquisition system, comprising:

[0035] Step 1: Collect traffic through the acquisition system and upload it to the traffic processing server in the form of traffic packets;

[0036] Step 2: Periodically perform operations on traffic packets using strategies such as removing invalid columns, merging random ports with the same IP address, filtering abnormal data, and labeling data. The processed traffic data is obtained through data cleaning and optimization deduplication.

[0037] Step 3: Set triggers to divide traffic data into risky data and non-risky data. The risky data is used for abnormal traffic investigation. The risky data and non-risky data are stored in the risky data record table. The triggers include: trigger conditions for a set time period, trigger conditions for a set IP address, trigger conditions for abnormal access, and trigger conditions for a set number of accesses.

[0038] Step 4: Extract risk data and non-risk data according to the set strategy, and then de-identify them to obtain de-identified data;

[0039] Step 5: Create a column named "warn_mes" with a boolean type in the risk data record table. Set the value of sent risk data to 1 and unsent risk data to 0. The default value of the column is 0. Periodically extract all risk data with a "warn_mes" value of 0 from the risk data record table and set the corresponding value of the extracted risk data to 1. For unsent risk data, first, form a complete abnormal data file for it to be provided to the first designated personnel. Then, distribute the de-identified risk data to the corresponding second designated personnel according to the source and target IP of the risk data.

[0040] Based on the same inventive concept, this application also provides an apparatus corresponding to the method in Embodiment 1, as detailed in Embodiment 2.

[0041] Example 2

[0042] like Figure 2 As shown, this embodiment provides a traffic optimization and analysis device for a data acquisition system, comprising:

[0043] The upload module collects traffic through the acquisition system and uploads it to the traffic processing server in the form of traffic packets;

[0044] The import module periodically operates on traffic packets using strategies such as removing invalid columns, merging random ports with the same IP address, filtering abnormal data, and labeling data. It also obtains processed traffic data through data cleaning and optimization deduplication.

[0045] The data mining module sets triggers to divide traffic data into risky data and non-risky data. The risky data is used for abnormal traffic investigation. The risky data and non-risky data are stored in a risky data record table. The triggers include: trigger conditions for a set time period, trigger conditions for a set IP address, trigger conditions for abnormal access, and trigger conditions for a set number of accesses.

[0046] The extraction and desensitization module extracts risky and non-risky data according to the set strategy, and then desensitizes the data to obtain desensitized data.

[0047] The risk warning forwarding module creates a column named "warn_mes" of type bool in the risk data record table, setting 1 for risk data that has been sent and 0 for those that have not been sent, with a default value of 0. It periodically extracts all risk data where "warn_mes" is 0 from the risk data record table and sets the corresponding value of the extracted risk data to 1. For unsent risk data, it first creates a complete abnormal data file, which is then provided to a first designated person. Secondly, it distributes the de-identified risk data to the corresponding second designated person based on the source and target IP of the risk data.

[0048] Since the apparatus described in Embodiment 2 of the present invention is an apparatus used to implement the method of Embodiment 1 of the present invention, those skilled in the art can understand the specific structure and variations of the apparatus based on the method described in Embodiment 1 of the present invention, and therefore will not be described again here. All apparatuses used in the method of Embodiment 1 of the present invention fall within the scope of protection of the present invention.

[0049] Based on the same inventive concept, this application provides an electronic device embodiment corresponding to Embodiment 1, as detailed in Embodiment 3.

[0050] Example 3

[0051] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it can implement any of the implementation methods in Embodiment 1.

[0052] Since the electronic device described in this embodiment is the device used to implement the method in Embodiment 1 of this application, those skilled in the art can understand the specific implementation method and various variations of the electronic device in this embodiment based on the method described in Embodiment 1 of this application. Therefore, how the electronic device implements the method in the embodiment of this application will not be described in detail here. Any device used by those skilled in the art to implement the method in the embodiment of this application falls within the scope of protection of this application.

[0053] Based on the same inventive concept, this application provides a storage medium corresponding to Embodiment 1, as detailed in Embodiment 4.

[0054] Example 4

[0055] This embodiment provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it can implement any of the implementation methods in Embodiment 1.

[0056] Example 5

[0057] like Figure 3 As shown, this embodiment describes a specific implementation method for the Colasoft flow acquisition system:

[0058] The Colasoft traffic acquisition system collects traffic and uploads it to the traffic processing server in the form of large traffic packets, which can support other acquisition systems.

[0059] Large traffic packets are periodically and automatically split, optimized, and deduplicated according to a strategy. The processed results are then imported into an SQLite database for traffic analysis and mining. Addressing the issue of large, information-heavy, and poorly readable data packets collected by Colasoft, strategies such as removing invalid columns, merging random ports from the same IP address, filtering out abnormal data, and data labeling are employed to clean and optimize the large data packets, resulting in clear and easily readable data for analysis and mining.

[0060] Traffic data is extracted based on user-defined policies, and sensitive information is anonymized to prevent leakage of sensitive information while meeting user needs for traffic data. Users can configure custom policies according to their needs, such as traffic information of specific source or target IP ranges or traffic information during sensitive periods. After the initial configuration, the system will periodically extract traffic that meets the conditions and generate files based on the policies. Since all traffic information in the extracted files is in plaintext, which could easily lead to information leakage, sensitive information is anonymized after the extracted files are generated to prevent leakage of sensitive information, ultimately achieving the goal of meeting user needs for traffic data while preventing leakage of sensitive information.

[0061] Customized policy triggers are designed to uncover potential risks from traffic data. Detected risky traffic is extracted and saved, and then the abnormal traffic data file is distributed to the network department and relevant personnel matched based on the problematic IP address for joint investigation. When traffic is cleaned and imported into the database, it undergoes a check by policy triggers. Traffic that triggers the policy conditions is recorded in the "Risk Data Record Table" for relevant personnel to investigate abnormal traffic. Customized policy triggers include trigger conditions such as specific time periods, specific IPs (obtained through whitelisting), abnormal access, and frequent access. The system can update in real time for newly added trigger conditions. An "alarm deduplication method" is designed and implemented. A column named "warn_mes" of type boolean is created in the "Risk Data Record Table," setting 1 for abnormal traffic that has already been sent and 0 for unsent traffic. The column's default value is 0. Periodically, all abnormal traffic with a "warn_mes" value of 0 in the "Risk Data Record Table" is extracted and distributed. The "warn_mes" value of the distributed abnormal traffic is then set back to 1 to prevent duplicate alarm messages from affecting subsequent investigations. For abnormal traffic that did not trigger an alarm, firstly, a complete abnormal traffic file was created and provided to the network department. Secondly, the source and destination IPs of the abnormal traffic were distributed to the relevant responsible persons according to the "IP Responsibility Map". Abnormal traffic was deduplicated using an alarm method, i.e., in a minimal visibility manner (abnormal traffic not under one's responsibility was not visible), to generate an abnormal traffic file and distribute it to the relevant responsible persons to assist the network department in investigating abnormal traffic.

[0062] It runs daily on a set schedule, requiring no manual intervention, and automatically analyzes and detects abnormal traffic. Furthermore, user-defined filtering strategies and customized policy triggers can be modified to expand its applicability.

[0063] By following the steps outlined above, we can address the challenges of analyzing large traffic packets and the risk of information leakage, while simultaneously enabling further data mining to create a complete, customized automated data collection and analysis system.

[0064] like Figure 4 As shown, the complete automated data acquisition and analysis system is packaged and packaged, and a Python-based GUI interface is developed to improve the interactivity between the analysis system and the user.

[0065] After processing, traffic data is first categorized by triggers (risky or non-risk data) before being imported into the database. This involves two operations: first, risky data is forwarded to network operations (data not anonymized) and application administrators (data anonymized); second, data is extracted according to user-defined policies, and then anonymized to ensure information security. It's not that triggers are set to extract risky data from the anonymized data; the imported data is not anonymized, but rather anonymized during the export process.

[0066] The technical solutions provided in the embodiments of this application have at least the following technical effects or advantages: the methods, apparatus, devices, and media provided in the embodiments of this application,

[0067] 1. To address the issue that data acquisition systems export large packages containing a lot of unorganized and poorly readable information, this invention automates the breakdown, optimization, and deduplication of these large packages, transforming them into highly readable data sets.

[0068] 2. Regarding the issue that data collection systems collect data traffic in plaintext and provide it to users, which can easily lead to the leakage of sensitive information, this invention performs special field anonymization operations on files distributed to viewers. This hides sensitive information and prevents information leakage without affecting user analysis.

[0069] 3. Compared with traditional Excel analysis methods, the present invention does not require manual intervention during operation, can stably process massive amounts of data, and does not need to worry about software crashes or slow analysis speeds due to software data storage limits. It frees up manpower, improves analysis efficiency, and expands the applicable scenarios.

[0070] 4. Compared with traditional methods, the custom strategies and strategy triggers of this invention are pre-designed and can be flexibly modified. Data traffic analysis is accurate, eliminating the possibility of missed analysis due to human error. Furthermore, all steps are automated, and detected abnormal traffic is automatically sent to relevant personnel, saving manual processing time while uncovering the potential value behind network traffic. Combined with the data acquisition system, it forms a complete customized automated data acquisition and analysis system.

[0071] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0072] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0073] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0074] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0075] While specific embodiments of the present invention have been described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and not intended to limit the scope of the present invention. Equivalent modifications and variations made by those skilled in the art in accordance with the spirit of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A flow optimization analysis method for a harvesting system, characterized by, The application comprises the following steps: Step 1, collecting traffic through a collection system and uploading traffic packets to a traffic processing server; Step 2, periodically operating the traffic packets by removing invalid columns, merging random ports of the same IP, screening abnormal data, and data labeling strategies, and obtaining processed traffic data through data cleaning and optimized deduplication; Step 3, setting a trigger to divide the traffic data into risk data and non-risk data, wherein the risk data is used for abnormal traffic investigation; The risk data and non-risk data are stored in a risk data record table; The trigger comprises a set period trigger condition, a set IP trigger condition, an abnormal access trigger condition, and a set number of access trigger conditions; Step 4, extracting and desensitizing the risk data and non-risk data according to a set strategy to obtain desensitized data; Step 5, forwarding the risk data to a first setting personnel and forwarding the desensitized risk data to a second setting personnel.

2. A flow optimization analysis method for a harvesting system according to claim 1, characterized in that, The step 5 is further specified as follows: a column named warn_mes of bool type is created in the risk data record table, the risk data that has been sent is set to 1, and the risk data that has not been sent is set to 0, and the default value of the column is 0; all risk data with warn_mes of 0 in the risk data record table is extracted, and the corresponding value of the extracted risk data is set to 1; for the risk data that has not been sent, firstly, the risk data is formed into a complete abnormal data file for providing the first setting personnel; secondly, the desensitized risk data is distributed to the corresponding second setting personnel according to the source and target IP of the risk data.

3. A flow optimization analysis device for a collection system, characterized by, The application comprises the following steps: An uploading module collects traffic through a collection system and uploads traffic packets to a traffic processing server; An importing module periodically operates the traffic packets by removing invalid columns, merging random ports of the same IP, screening abnormal data, and data labeling strategies, and obtains processed traffic data through data cleaning and optimized deduplication; A mining module sets a trigger to divide the traffic data into risk data and non-risk data, wherein the risk data is used for abnormal traffic investigation; The risk data and non-risk data are stored in a risk data record table; The trigger comprises a set period trigger condition, a set IP trigger condition, an abnormal access trigger condition, and a set number of access trigger conditions; An extraction and desensitization module extracts and desensitizes the risk data and non-risk data according to a set strategy to obtain desensitized data; A risk warning forwarding module forwards the risk data to a first setting personnel and forwards the desensitized risk data to a second setting personnel.

4. A flow optimization analysis device for a harvesting system according to claim 3, characterized in that, The risk early warning forwarding module is further specifically: creating a column named warn_mes in the risk data record table, the column being of bool type, setting 1 for the risk data that has been sent and 0 for the risk data that has not been sent, and the default value of the column being 0; timing extracting all the risk data whose warn_mes in the risk data record table is 0, and setting the corresponding value of the extracted risk data to 1; for the risk data that has not been sent, firstly forming a complete abnormal data file for providing the first setting personnel; secondly distributing the desensitized risk data to the corresponding second setting personnel according to the source and target IP of the risk data.

5. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the method of any one of claims 1 and 2 when executing the program.

6. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method of any one of claims 1 and 2.

Images