Data processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a data processing method and device, electronic equipment and a storage medium, and aims to load data into a Hadoop file and store the data according to columns, so that a column index-oriented storage mode can be established in Redis, and the storage performance of the Redis can be optimized. According to the embodiment of the invention, the access frequency of each key value in the Redis can be adjusted, and it is determined whether the storage mode of the data in the Redis is the data or the access address of the data in the Hadoop file, thereby further optimizing the storage space of the Redis and solving the problem that the data cannot be put into the Redis memory due to too large data volume. Besides, according to the embodiment of the invention, the hotspot data can be left in the Redis according to the access frequency of each key value in the Redis, and the cold data is placed on the Hadoop, so that the data access efficiency can be improved.

Description

technical field [0001] The present invention relates to the field of computer technology, in particular to a data processing method, device, electronic equipment and storage medium. Background technique [0002] Today, Hadoop has become the main tool for enterprises to process big data due to its high efficiency and reliability in processing data. However, due to the parallel batch processing characteristics of Hadoop, Hadoop cannot effectively adapt to the processing of shared data, such as the creation of tree indexes, the training of iterative algorithms such as PageRank, etc. There are currently the following solutions to this problem: [0003] The first is a Hadoop-based file system. HDFS is responsible for storing and managing data in Hadoop. HDFS is the Hadoop distributed file system. It is a master / slave structure and consists of a NameNode node and multiple DataNode nodes. Data in HDFS is stored in various DataNodes (data nodes) of HDFS. For the shared data, it...

AI Technical Summary

Problems solved by technology

[0007] First, the problem of excessive data volume

The amount of shared data in practical applications may be very large. When the amount of data is very large, for example, when there are about 1 billion records per month, it is not suitable for Hadoop distributed cache. Cannot be stored in memory, resulting in inability to process

[0008] Secondly, data access efficiency issues

For the situation where the amount of shared data is too large, if the Hadoop file system is used directly, a large number of disk I / O operations will be generated, thereby reducing the efficiency of data processing; in addition, if the data has many dimensions in the form of row storage (that is, the data There are a lot of columns) but very few are actually used, and a large amount of useless data will be read, which will reduce the efficiency of data transmission and processing. Even if Redis is used for processing, Redis will become the bottleneck of system processing

Images