Machine data storage method and system

A machine data and data writing technology, which is applied in the direction of electrical digital data processing, special data processing applications, instruments, etc., can solve the problems of huge machine data volume, increased system overhead, data injection accumulation, etc., to reduce additional overhead, Ensure high-speed access and improve storage efficiency

Active Publication Date: 2017-11-07
昆仑智汇数据科技(北京)有限公司
View PDF4 Cites 4 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] However, due to the large volume of machine data, for 7*24 hours of continuous high-throughput machine data, if the existing MapReduce computing framework (MapReduce is a programming model for large-scale data sets (greater than 1TB) Parallel computing) or traditional database transaction methods, ACID (ACID principles are four principles for the normal execution of database transactions, referring to atomicity, consistency, independence and persistence) verification on large-scale data processing, such as data If the storage is abnormal, the large batch of data needs to be re-stored, which not only increases the overhead of the system, but also may lead to the problem of data injection accumulation

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Machine data storage method and system
  • Machine data storage method and system
  • Machine data storage method and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0056]Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided for more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0057] Those skilled in the art can understand that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art to which this invention belongs. It should also be understood that terms, such as those defined in commonly used dictionaries, should be understood to have meanings consistent with the meanings in the con...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a machine data storage method and system. The method comprises steps of dividing machine data into a plurality of data recording sets storing data according to batch orders, building a first preset mark in a current-batch data pre-writing log, conducting data pre-process to the current-batch data recording set via a parallel way, updating a state mark of the first preset mark according to a processing result, building a second preset mark for each data recording set in the current-batch data pre-writing log if the state mark of the first preset mark is a success mark, loading the current-batch data recording set to a corresponding distributed storage device via a parallel way, and updating a state mark corresponding to the second preset mark according to a data loading result of each data recording set. In the machine data storage method and system, a data recording set as a unit is marked, so data can be restored with the data recording set as granularity during post-operation; introduction of part of incomplete data can be avoided and system extra expenses can be reduced.

Description

technical field [0001] The invention relates to the technical field of machine data processing, in particular to a machine data storage method and system. Background technique [0002] With the wide application of Internet technology, in the context of the technology industry and the combination of traditional industries and the Internet industry, the growth of enterprise machine data is exponential. These data contain valuable value, and when machine data is accessed, it needs Enable reliable data storage. [0003] However, due to the large volume of machine data, for 7*24 hours of continuous high-throughput machine data, if the existing MapReduce computing framework (MapReduce is a programming model for large-scale data sets (greater than 1TB) Parallel computing) or traditional database transaction methods, ACID (ACID principles are four principles for the normal execution of database transactions, referring to atomicity, consistency, independence and persistence) verific...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
CPCG06F16/2219G06F16/258
Inventor 张硕程明史巨伟王晨
Owner 昆仑智汇数据科技(北京)有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap