Intelligence data processing method and apparatus

Abstract

The present invention provides an intelligence data processing method and apparatus. The foregoing method comprises the following steps: acquiring a network transmission rate, a disk read / write rate, a CPU utilization rate, and separately comparing the network transmission rate, the disk read / write rate, and the CPU utilization rate with a first preset value, a second preset value, and a third preset value, to acquire a comparison result; and controlling a hardware compression card to process data based on a preset control policy according to the comparison result. Decompression processing is performed on data by using an independent hardware compression card. Therefore, computational burden of a CPU is reduced, so that the CPU may focus on processing an operation related task, thereby improving execution efficiency of big data computation.

Description

technical field [0001] The invention belongs to the field of data processing, in particular to an intelligent data processing method and device. Background technique [0002] As human society enters the information age, data has become an equally important strategic resource like water and oil. By mining massive amounts of data, the operational decisions of governments and enterprises can be based on a more scientific basis, improving decision-making efficiency, crisis response capabilities, and public service levels. [0003] Although big data is extremely valuable, due to its huge scale, it faces the problem of continuous scalability. For example, the historical status data commonly used in data warehouses to record the behavior of an object, in the era of big data, with the rapid increase in the number of objects (such as customers) and the rapid increase in the behavior of each processing object (such as customer transactions), the scale becomes Explosive growth has br...

AI Technical Summary

Problems solved by technology

[0004] In response to the above requirements, the current mainstream big data processing systems, such as Hadoop, Hbase, Hive, etc., all support compression methods such as Gzip, LZO, and Snappy in applications, but no matter what method is adopted, the essence is to use CPU runtime to process data. The encoding, compression and decompression of the computer will inevitably have an impact on the operating efficiency of the CPU

[0005] In response to the above problems, the industry proposes to use hardware compression card technology, and use the computer chip on the hardware compression card to independently compress and decompress data. However, for tasks with low CPU and hard disk loads, using a hardware compression card is equivalent to additional The data undergoes a process of compression and decompression, and the resulting processing delay time is often higher than the saved IO time

Images