Unlock instant, AI-driven research and patent intelligence for your innovation.

Unstructured event log data classification and storage method and device

An unstructured, data storage technology, applied in the direction of unstructured text data retrieval, text database clustering/classification, electronic digital data processing, etc., can solve the problems affecting the amount of information, retrieval efficiency and comprehensive impact, and achieve Easy to filter effects

Inactive Publication Date: 2016-10-26
ANHUI TIANSHU INFORMATION TECH CO LTD
View PDF4 Cites 8 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

The division and storage of unstructured data has always been a difficult problem, which not only affects the amount of stored information and storage costs, but also has a great impact on the subsequent retrieval efficiency and comprehensiveness

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Unstructured event log data classification and storage method and device
  • Unstructured event log data classification and storage method and device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0024] First, the data information of different locations and sensors is collected through the data acquisition module, and then passed to the data classification module through the data transmission module. According to the time and coordinate information of the event data, the space is firstly divided in the form of a grid. Each grid To correspond to a square area in geographical space, arrange them in chronological order, and finally store the divided data in the data storage module. The data storage module uses 8MB data blocks to store them in slices, and divides each shard into a series of Segment, each Segment contains a series of Events, and then extracts a specific domain Field for the Event, and performs word segmentation on the entire Event information, and finally creates a full-text index to realize data filtering, data conversion, data grouping, and data aggregation processing, which is convenient Subsequent data retrieval.

Embodiment 2

[0026] First, the data information of different locations and sensors is collected through the data acquisition module, and then passed to the data classification module through the data transmission module. According to the time and coordinate information of the event data, the space is firstly divided in the form of a grid. Each grid In order to correspond to a square area of ​​geographical space, it is arranged in sequence according to time, and finally the divided data is stored in the data storage module. The data storage module uses a 16MB data block and adopts a fragmented storage method to divide each shard into a series of Segment, each Segment contains a series of Events, and then extracts a specific domain Field for the Event, and performs word segmentation on the entire Event information, and finally creates a full-text index to realize data filtering, data conversion, data grouping, and data aggregation processing, which is convenient Subsequent data retrieval.

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides an unstructured event log data classification and storage method and device. The storage device comprises a data collection module, a data transfer module, a data classification module and a data storage module. The method comprises a data collection and transfer step of collecting information data collected by different sensors at different places; a data classification step of classifying the collected information data based on a time and space mode, and storing the classified data in the storage module; and a data storage step of storing the data through adoption of a segment mode. According to the method and the device, classified storage is carried out on the different event information described by different sensors at different places, and the follow-up data extraction, statistics and analysis can be accelerated.

Description

technical field [0001] The invention relates to a data storage technology, in particular to a method and device for dividing and storing unstructured event log data. Background technique [0002] With the development of network technology, especially the rapid development of Internet and Intranet technology, the amount of unstructured data is increasing day by day. At this time, the limitations of relational databases, which are mainly used to manage structured data, become more and more obvious. Therefore, database technology has entered the "post-relational database era" accordingly, and has developed into the era of unstructured databases based on network applications. The rapid development of unstructured data is a big challenge to the storage capacity space. The multi-storage system of unstructured data not only has the characteristics of strong fault tolerance, high availability and scalability in terms of storage capacity, but also can utilize different types of The...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
CPCG06F16/335G06F16/316G06F16/35
Inventor 陈凌岳
Owner ANHUI TIANSHU INFORMATION TECH CO LTD