Mass data retrieval method based on DPI (Deep Packet Inspection) call ticket

Abstract

The invention relates to a mass data retrieval method based on a DPI call ticket, and belongs to the technical field of DPI data retrieval. Comprising the following steps: S1, data collection: identifying a collected data stream by adopting a DPI, and then carrying out format processing on the data stream to obtain collected DPI ticket data; s2, DPI call ticket data processing: cleaning, sorting and summarizing the collected DPI call ticket data, and finally establishing a DPI call ticket data storage library through processing of a data server; s3, key words are established, a UserAgent field is obtained from the collected DPI data, word segmentation is carried out on the UserAgent field, and one or more key words are obtained; s4, establishing a keyword database; s5, performing data retrieval; s6, data presentation; and S7, analyzing and summarizing the data. According to the mass data retrieval method based on the DPI call ticket, mass DPI call ticket data can be effectively processed and retrieved, efficient query service and deep analysis and insight are provided, and retrieval efficiency is greatly improved.

Description

Technical Field

[0001] The invention relates to a mass data retrieval method based on a DPI call list, belonging to the technical field of DPI data retrieval. Background Art

[0002] DPI (Deep Packet Inspection) call data usually refers to network communication data collected through network detection technology. The massive data retrieval method of DPI call data mainly relies on deep data packet inspection (DPI) technology, which analyzes network traffic data to achieve in-depth insight into network behavior and efficient retrieval.

[0003] Currently, when searching for massive amounts of DPI call bill data, the most common method is to obtain a massive database of DPI call bills, and then enter keywords to query and retrieve corresponding data from the massive database of DPI call bills. This retrieval method is often difficult to effectively process and retrieve massive amounts of DPI call bill data, and it is difficult to provide efficient query services and in-depth analytical insights, and the retrieval efficiency needs to be improved.

[0004] Based on this, the present invention provides a method for retrieving massive data based on DPI call records. Summary of the invention

[0005] In view of this, the present invention provides a massive data retrieval method based on DPI call records, which can effectively process and retrieve massive DPI call record data, provide efficient query services and in-depth analysis insights, and greatly improve retrieval efficiency.

[0006] The present invention provides a method for retrieving massive data based on DPI call records, and the proposed technical solution comprises the following steps:

[0007] S1: Data collection: Use DPI to identify the collected data stream, and then format it to obtain the collected DPI call record data;

[0008] S2: DPI call bill data processing: clean, organize and summarize the collected DPI call bill data, and finally establish a DPI call bill data repository through data server processing;

[0009] S3: Establish keywords: obtain the UserAgent field from the collected DPI data, segment the UserAgent field, and obtain one or more keywords;

[0010] S4: Establishing a keyword database: Filtering a corresponding keyword database from the DPI call record data repository in step S2 according to the corresponding one or more keywords;

[0011] S5: Data retrieval: The user enters a data keyword in the network terminal, and the request sent is processed by the data server, and the corresponding data is retrieved from the DPI call data storage repository in step S2 and the keyword database in step S4;

[0012] S6: Data presentation: sending the retrieved corresponding data query results back to the network terminal;

[0013] S7: Data analysis and summary: Analyze the data retrieved in step S6 and select the required data for summary.

[0014] Furthermore, in step S1: massive network traffic information is collected through network detection technology to form a data stream, the data stream is identified using DPI, and the identified DPI data is obtained, the DPI data is formatted and the collected DPI call record data is obtained.

[0015] Furthermore, in step S2: the data server compresses the massive data to reduce storage requirements and speed up retrieval. The data compression method adopts one of the RLE compression algorithm and the Lempel-Ziv compression algorithm. Spark is used to process the data in parallel, and the retrieval task can be distributed to multiple computing nodes to improve the retrieval efficiency.

[0016] Furthermore, in step S3: a regular expression class is used to segment the UerAgent field.

[0017] Furthermore, in step S1: the DPI call record data includes the source and destination addresses, timestamp, protocol type, and region of the communication; in step S3: the keywords include the source and destination addresses, timestamp, protocol type, and region of the communication.

[0018] Furthermore, in step S4: the input data keywords are searched and recorded, so that the user can perform a second quick search on the query results based on the search history and browsing history.

[0019] Furthermore, in step S5: a search order is determined according to the DPI call record data storage repository and the keyword database corresponding to the input data keyword, thereby quickly acquiring the database corresponding to the data keyword.

[0020] Furthermore, in step S6: weight analysis and sorting are performed on the retrieved corresponding data query results, and the query results with higher weights are placed in the front.

[0021] Further, in step S7: the data is summarized through PPT documents, Excel tables, and Word documents to transform the data into understandable and actionable insights to provide support and suggestions to users.

[0022] Beneficial effects of the present invention:

[0023] The present invention can effectively process and retrieve massive DPI call bill data through the massive data retrieval method based on the DPI call bill, provide efficient query services and in-depth analysis insights, and greatly improve the retrieval efficiency. DETAILED DESCRIPTION

[0024] The preferred embodiments of the present invention will be described in detail below.

[0025] The present invention provides a method for retrieving mass data based on DPI call records, which comprises the following steps:

[0026] S1: Data collection: Collect massive network traffic information through network detection technology to form data streams, use DPI to identify the data streams, and obtain the identified DPI data, convert the DPI data into a format, and obtain the collected DPI call data. DPI equipment or software usually includes traffic capture and analysis functions;

[0027] S2: DPI call data processing: Clean, organize and summarize the collected DPI call data, use data cleaning tools to remove invalid, duplicate or erroneous data records, use ETL extraction, transformation and loading processes to format the data to conform to the standard format, and finally establish a DPI call data repository through data server processing;

[0028] S3: Establish keywords: obtain the UserAgent field from the collected DPI data, segment the UserAgent field, and obtain one or more keywords;

[0029] S4: Establishing a keyword database: Filtering a corresponding keyword database from the DPI call record data repository in step S2 according to the corresponding one or more keywords;

[0030] S5: Data retrieval: Use one of the search engine technology and query optimization technology to improve retrieval efficiency and accuracy. The user enters keywords for data retrieval. The retrieval request issued by the user is processed by the data server, and the DPI call data repository and keyword database are queried to retrieve relevant data.

[0031] S6: Data presentation: using one of the Web front-end technology and data visualization tools to display the search results, and sending the retrieved corresponding data query results back to the network terminal.

[0032] S7: Data analysis and summary: Analyze the data retrieved in step S6 and select the required data for summary.

[0033] As an optional embodiment, in step S2: the data server compresses massive data to reduce storage requirements and speed up retrieval, the data compression method adopts one of the RLE compression algorithm and the Lempel-Ziv compression algorithm, and uses Spark to process the data in parallel, and the retrieval task can be distributed to multiple computing nodes to improve the retrieval efficiency.

[0034] As an optional embodiment, in step S3: a regular expression class is used to segment the UerAgent field.

[0035] As an optional embodiment, in step S1: the DPI call record data includes the source and destination addresses, timestamp, protocol type, and region of the communication; in step S3: the keywords include the source and destination addresses, timestamp, protocol type, and region of the communication.

[0036] As an optional embodiment, in step S4: the input data keywords are searched and recorded, so that the user can perform a second quick search on the query results based on the search history and browsing history.

[0037] As an optional embodiment, in step S5: determine the search order according to the DPI call record data storage repository and the keyword database corresponding to the input data keyword, and then quickly obtain the database corresponding to the data keyword.

[0038] As an optional embodiment, in step S6: the retrieved corresponding data query results are weighted and sorted, and the query results with higher weights are placed in the front.

[0039] As an optional embodiment, in step S7: data is summarized through PPT documents, Excel tables, and Word documents to convert the data into understandable and actionable insights to provide support and suggestions for users.

[0040] Working principle:

[0041] S1: Data collection: Collect massive network traffic information through network detection technology to form data streams, use DPI to identify the data streams, and obtain the identified DPI data, convert the DPI data into a format, and obtain the collected DPI call list data. DPI equipment or software usually includes traffic capture and analysis functions. DPI call list data includes the source and destination addresses, timestamps, protocol types, and regions of the communication. S2: DPI call list data processing: Clean, organize and summarize the collected DPI call list data, use data cleaning tools to remove invalid, duplicate or erroneous data records, and use ETL extraction, conversion, and loading processes to format the data. To make it conform to the standard format, the data server compresses the massive data to reduce storage requirements and speed up retrieval. The data compression method adopts one of the RLE compression algorithm and Lempel-Ziv compression algorithm. Spark is used to process the data in parallel, and the retrieval task can be distributed to multiple computing nodes to improve the retrieval efficiency. The DPI call data repository is finally established through data server processing; S3: Establish keywords: Get the UserAgent field from the collected DPI data, use the regular expression class to segment the UerAgent field, and obtain one or more keywords. The keywords include the source and destination addresses, timestamps, and protocols of the communication. Type, region; S4: Establish keyword database: Filter the corresponding keyword database from the DPI call data repository in step S2 according to the corresponding one or more keywords, mark the input data keywords for retrieval and record them, so that users can perform secondary fast retrieval of the query results based on search history and browsing history; S5: Data retrieval: Use one of the search engine technology and query optimization technology to improve retrieval efficiency and accuracy. Users enter keywords for data retrieval. The retrieval request issued by the user is processed by the data server, and the DPI call data repository and keyword database are queried to retrieve relevant data. The DPI call data repository and keyword database determine the search order, and then quickly obtain the database corresponding to the data keyword; S6: Data presentation: Use one of the Web front-end technology and data visualization tools to display the search results, and send the retrieved corresponding data query results back to the network terminal, and perform weight analysis and sorting on the retrieved corresponding data query results, and put the query results with higher weights in the front row; S7: Data analysis summary: Summarize data through PPT documents, Excel tables, and Word documents, select the required data for summary, so as to convert the data into understandable and actionable insights, and provide support and suggestions for users.

[0042] The present invention can effectively process and retrieve massive DPI call bill data through the massive data retrieval method based on the DPI call bill, provide efficient query services and in-depth analysis insights, and greatly improve the retrieval efficiency.

[0043] The present invention and its implementation methods are described above, and such description is not restrictive, and the actual structure is not limited thereto. In short, if ordinary technicians in the field are inspired by it, without departing from the purpose of the invention, they can design structures and embodiments similar to the technical solution without creativity, which should all fall within the protection scope of the present invention.

Claims

1. A method for retrieving massive data based on DPI call records, characterized by: The following steps are involved: S1: Data collection: Use DPI to identify the collected data stream, and then format it to obtain the collected DPI call record data; S2: DPI call bill data processing: clean, organize and summarize the collected DPI call bill data, and finally establish a DPI call bill data repository through data server processing; S3: Establish keywords: obtain the UserAgent field from the collected DPI data, segment the UserAgent field, and obtain one or more keywords; S4: Establishing a keyword database: Filtering a corresponding keyword database from the DPI call record data repository in step S2 according to the corresponding one or more keywords; S5: Data retrieval: The user enters a data keyword in the network terminal, and the request sent is processed by the data server, and the corresponding data is retrieved from the DPI call data storage repository in step S2 and the keyword database in step S4; S6: Data presentation: sending the retrieved corresponding data query results back to the network terminal; S7: Data analysis and summary: Analyze the data retrieved in step S6 and select the required data for summary.

2. A method for retrieving massive data based on DPI call records according to claim 1, characterized in that: In step S1: massive network traffic information is collected through network detection technology to form a data stream, DPI is used to identify the data stream, and the identified DPI data is obtained, the DPI data is formatted and the collected DPI call record data is obtained.

3. A method for retrieving massive data based on DPI call records according to claim 1, characterized in that: In step S2: the data server compresses the massive data to reduce storage requirements and speed up retrieval. The data compression method adopts one of the RLE compression algorithm and the Lempel-Ziv compression algorithm. Spark is used to process the data in parallel, and the retrieval task can be distributed to multiple computing nodes to improve the retrieval efficiency.

4. The method for retrieving massive data based on DPI call records according to claim 1, characterized in that: In step S3: a regular expression class is used to segment the UerAgent field.

5. The method for retrieving mass data based on DPI call records according to claim 1, characterized in that: In step S1: the DPI call record data includes the source and destination addresses, timestamp, protocol type, and region of the communication; in step S3: the keywords include the source and destination addresses, timestamp, protocol type, and region of the communication.

6. A method for retrieving massive data based on DPI call records according to claim 1, characterized in that: In step S4: the input data keywords are searched and recorded, so that the user can perform a second quick search on the query results based on the search history and browsing history.

7. A method for retrieving mass data based on DPI call records according to claim 1, characterized in that: In step S5: the search order is determined according to the DPI call record data storage library and the keyword database corresponding to the input data keyword, so as to quickly obtain the database corresponding to the data keyword.

8. A method for retrieving mass data based on DPI call records according to claim 1, characterized in that: In step S6: the retrieved corresponding data query results are weighted and sorted, and the query results with higher weights are placed in the front.

9. A method for retrieving mass data based on DPI call records according to claim 1, characterized in that: In step S7: Summarize the data through PPT documents, Excel tables, and Word documents to transform the data into understandable and actionable insights to provide support and suggestions to users.