Unlock instant, AI-driven research and patent intelligence for your innovation.

Data compression method and device based on big data and medium

A data compression and big data technology, applied in the field of data processing, can solve the problems of occupying a large storage space, short collection interval, huge amount of index data, etc., and achieve the effect of reducing the waste of storage space

Active Publication Date: 2020-08-11
TRANSWARP INFORMATION TECH SHANGHAI
View PDF7 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] Due to the large number of data collection points and short collection intervals, the data volume of real-time monitoring index data is very large
When there are many system nodes and many indicator definitions, the indicator data will have a huge amount of data and will occupy a huge amount of storage space

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data compression method and device based on big data and medium
  • Data compression method and device based on big data and medium
  • Data compression method and device based on big data and medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0055] Figure 1e This is a flowchart of a data compression method based on big data provided in the first embodiment of the present invention. The embodiment of the present invention is applicable to the case of compressing and storing index data. The method can be executed by the data compression device based on big data provided by the embodiment of the present invention. The device can be implemented by software and / or hardware. Can be integrated in computer equipment. Such as Figure 1e As shown, the method of the embodiment of the present invention specifically includes:

[0056] Step 101: When a big data compression request is detected, obtain historical index data to be compressed corresponding to the big data decision request.

[0057] The historical indicator data to be compressed includes multiple indicator data sets in units of days. The indicator data set includes all indicator data collected on a corresponding date. The indicator data in the indicator data set are ar...

Embodiment 2

[0103] figure 2 This is a flowchart of a data compression method based on big data provided in the second embodiment of the present invention. The embodiment of the present invention can be combined with each of the optional solutions in one or more of the above embodiments. In the embodiment of the present invention, according to multiple index data sets in units of days and preset data slice range size parameters, Obtaining the mode corresponding to the subscript of each data shard can include: randomly obtaining a set number of indicator data sets among multiple indicator data sets with a unit of day as the mode election sample; in the mode election sample For each indicator data set of, the indicator data in the indicator data set is sliced ​​according to the preset data slice range size parameter, and the indicator data in the indicator data set is divided into multiple indicator data slices, and in chronological order, Determine the data segment index corresponding to ea...

Embodiment 3

[0122] image 3 This is a schematic structural diagram of a data compression device based on big data provided in the third embodiment of the present invention. The device can be implemented in software and / or hardware, and can generally be integrated in computer equipment. Such as image 3 As shown, the device includes: a data acquisition module 301, a data fragmentation module 302, and a data compression module 303.

[0123] Among them, the data acquisition module 301 is used to acquire the historical index data to be compressed corresponding to the big data decision request when a big data compression request is detected. The historical index data to be compressed includes multiple index data sets in units of days. The data set includes all the indicator data collected on the corresponding date. The indicator data in the indicator data set are arranged in chronological order; the data slicing module 302 is used for multiple indicator data sets with a unit of day and preset dat...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The embodiment of the invention discloses a data compression method and device based on big data and a medium. The method comprises the steps of when a big data compression request is detected, acquiring to-be-compressed historical index data corresponding to the big data decision request, wherein the to-be-compressed historical index data comprises a plurality of index data sets with days as units, and the index data sets comprise all index data collected on the corresponding dates; obtaining a mode corresponding to each data fragment subscript according to the plurality of index data sets taking the day as the unit and a preset data fragment range size parameter; According to the mode corresponding to each data fragment subscript and the short-time memory cardinal number corresponding toeach index data set, determining a mark field of the index data in each index data set, and performing compression storage on the index data in each index data set. According to the embodiment of theinvention, the index data can be compressed and stored, and the index data is not lost while the waste of the storage space of the index data is reduced.

Description

Technical field [0001] The embodiments of the present invention relate to data processing technology, and in particular to a data compression method, device, and medium based on big data. Background technique [0002] A very important part of the Internet Technology (IT) operation and maintenance work is to monitor and record the operating status of each host device in the system and network load in real time, and obtain the index data of each host device to achieve abnormalities. Real-time warning, fault diagnosis and data mining functions. [0003] Due to the large number of data collection points and short collection intervals, the data volume of the indicator data for real-time monitoring is very large. When there are many system nodes and many indicators are defined, the indicator data will have a huge amount of data and will occupy a huge amount of storage space. Summary of the invention [0004] The embodiment of the present invention provides a data compression method, dev...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F11/30
CPCG06F11/3006G06F11/3024G06F11/3082
Inventor 黄南溪郭建新罗辉
Owner TRANSWARP INFORMATION TECH SHANGHAI