Network abstract data-based index creation method and system

Abstract

The invention provides a network abstract data-based index creation method and system. The method comprises the following steps of: generating a data block according to a real-time data flow capturedfrom an internet access; generating abstract data according to a key field in the data block; and generating index data according to the abstract data. According to the method and system, the problemof data loss caused by the reason that original data cannot be written into disks in time can be solved; the disk IO conflicts in index creation processes are reduced, the system throughput is improved, and real-time data can be written into the disks under big data flows; the problem that the data index creation is not timely is solved, only a small amount of abstract data needs to be read, and the disk overhead and CPU overhead during the index creation are reduced; the flexibility is good, and when the system load is low, index creation can be carried out by utilizing a plurality of CPUs; when the load is high, the degree of parallelism is reduced and resources are turned over to other modules; and data captured from internet accesses in real time can be stored into disks, and corresponding inverted index data can be generated in real time.

Description

technical field [0001] The invention belongs to the field of performance monitoring, and specifically relates to an index creation method and system based on network summary data. Background technique [0002] In the field of performance monitoring, it is necessary to capture the communication data between the client and the server from the network port in real time and save it to the disk to form the original data. At the same time, during the running process of performance monitoring, users have the requirement to query and extract some data from the data saved to the disk. [0003] In order to meet the needs of users to query and extract some data, the original data needs to be indexed. [0004] If you use an ordinary database, when inserting data with index fields, you need to go through the processes of "connecting to the database", "analyzing request statements", "inserting data records", "inserting indexes", and "closing connections". The whole process takes a long t...

AI Technical Summary

Problems solved by technology

If this patent document is used in the scene of real-time capture, storage and indexing in the present invention, the disadvantage is that the memory consumption is too large, causing the program to fail to run normally, or part of the data must be discarded

Images