Data reading method and device, electronic equipment and storage medium

Abstract

The invention relates to the field of big data, and discloses a data reading method and device, electronic equipment and a storage medium, and the method comprises the steps: receiving a data reading request transmitted by a client, converting the data reading request into a data query statement, constructing an index of the data query statement, and obtaining a target query statement; according to a query object in the target query statement, splitting the target query statement to obtain a plurality of sub-query statements, and reading query data of each sub-query statement from a source data party corresponding to the client to obtain a plurality of pieces of query data; performing relation integration on the plurality of query data to generate a query file of the target query statement; and fragmenting the query file to obtain a plurality of sub-fragmented files, and returning the plurality of sub-fragmented files to the local disk of the client. In addition, the invention also relates to a block chain technology, and the sub fragment files can be stored in a block chain. According to the invention, the data reading efficiency can be improved.

Description

technical field [0001] The present invention relates to the field of big data, in particular to a data reading method, device, electronic equipment and computer-readable storage medium. Background technique [0002] With the development of big data, data reading has become an indispensable process. The data reading can be understood as obtaining the data needed by the business side from the data source. For example, in the process of user location analysis, how to read efficiently It is particularly important to extract the user's label data to realize user analysis. [0003] At present, data reading is usually based on the big data platform as the calculation method to pull the required data from the data source, but this method is prone to single-node performance problems, that is, the data will be in a serial state during the reading process. In this way, it is easy to cause the local loading of data reading to take a very long time, which will affect the efficiency of d...

AI Technical Summary

Problems solved by technology

[0003] At present, data reading is usually based on the big data platform as the calculation method to pull the required data from the data source, but this method is prone to single-node performance problems, that is, the data will be in a serial state during the reading process. This will easily cause the local loading of data reading to take a very long time, which will affect the efficiency of data reading

Images