Method for establishing and applying aggregation type IP index

An application method and aggregated technology, applied in the field of data processing, can solve problems such as system space occupation and process complexity, achieve high performance, reduce memory and storage space, and reduce memory consumption

Active Publication Date: 2021-10-19
COLASOFT
View PDF11 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] The present invention aims to solve the problem of complex data storage and query process in the prior art and easy to cause system space occupation, and proposes a method for establishing and applying an aggregated IP index with good performance and high efficiency

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method for establishing and applying aggregation type IP index
  • Method for establishing and applying aggregation type IP index
  • Method for establishing and applying aggregation type IP index

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0040] This embodiment provides a method for establishing and applying an aggregated IP index, including the following steps:

[0041] Step S1, index design: convert all 4 bytes in the IP domain into integers corresponding to the host byte order, and create a blank bitmap index; design the IP bitmap in the host byte order, and retrieve it To improve the hit rate, the IP bit marks of the same network will be similar. When reading and matching similar IPs, the optimization effect will be achieved. Since most networks plan IPs in the form of network segments, similar IPs will appear in the network.

[0042] Step S2, index marking: receiving data blocks, extracting the IP data of all data blocks arriving in a space-time and converting them into integers, putting them into a blank bitmap index and marking the corresponding positions to form a local time-space processing index; using bitmaps Index mode, which can record all IP conditions without data distortion at all, and the bitma...

Embodiment 2

[0046] The present invention provides a method for establishing and applying an aggregated IP index, which is based on the following index design strategy:

[0047] Step 1. Convert all 4 bytes in the IP domain range (0-255.255.255.255) into integers corresponding to the host byte order, and the converted integers range from 0 to 4294967295.

[0048] Step 2: Create a mask bit for each integer, and a total of 4294967295 bits are required. These 4294967295 bits form a blank bitmap index, and a total of 512MB of memory is consumed to record the bitmap index.

[0049]Step 3. Divide the integers evenly by segment to form a slice array, and compile a slice sequence number (0-15) for each slice array; in this embodiment, divide the integers from 0 to 4294967295 by segment into 16 slices, 32MB each. The data size of each slice is 268435456, the data range of 0 slices is [0,268435456) or [0,268435455], the data range of 1 slice is [268435456, 536870911], and so on, the data range of 15...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a method for establishing and applying an aggregation type IP index, and belongs to the technical field of data processing. The method comprises the following steps: converting four bytes in all IP domain ranges into integers corresponding to a host byte order, and establishing a blank bitmap index; extracting IP data in all data blocks received in a space-time, converting the IP data into integers, putting the integers into corresponding positions in the blank bitmap index, and marking the integers to form a processing index; performing logical 'or' operation on the processing indexes and aggregating the processing indexes into larger spatio-temporal indexes; and receiving the IP data to be searched, determining whether the integer converted from the IP data to be searched to the corresponding host byte order is in the larger spatio-temporal index, and if so, reading the smaller spatio-temporal indexes in sequence until the IP data is positioned. Through adoption of the method provided by the invention, whether certain IP data exists in the original data within the effective time range of the data can be simply and quickly judged.

Description

technical field [0001] The invention relates to the field of data processing, in particular to a method for establishing and applying an aggregated IP index. Background technique [0002] The system collects network traffic according to time or space, and these network traffic contains a large amount of raw data. If it is necessary to determine whether there is a certain IP data in the traffic collected in each period, it is necessary to establish and retrieve the IP index, but due to the limitation of the system storage space, it is necessary to delete the IP data and the index. In the prior art, patent CN110825940A discloses a method for storing and querying network data packets. The technical solution is: when receiving network data packets, put the data packets into the data packet writing queue; analyze the characteristics of the data packets, And extract feature data; generate Bloom filter, bitmap index and offset array files according to feature data; write data pack...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & AuthorityApplications(China)
IPC IPC(8): G06F16/22G06F16/2455G06F16/951G06F16/9537H04L29/12
CPCG06F16/2237G06F16/2272G06F16/2282G06F16/2455G06F16/951G06F16/9537H04L2101/60
Inventor田红伟徐文勇王伟旭
OwnerCOLASOFT