Data processing method and device

Abstract

The invention discloses a data processing method and a data processing device and relates to the technical field of computers. A specific embodiment of the data processing method includes a step of uniformly converting multiple data sources including text data, relational database data and distributed cluster data into Spark Dataframe and registering as a Spark temporary table, and a step of inquiring the Spark temporary table across the data sources according to an input of a user in an sql mode. According to the embodiment, under the premise of retaining an original business logic, the memory and computational pressure of an original traditional database is transferred to a distributed cluster and is converted into bandwidth pressure, and the mixed query for multiple types of data sources is supported.

Description

technical field [0001] The present invention relates to the field of computer technology, and in particular to a data processing method, a data processing device, electronic equipment and a storage medium based on a distributed environment and memory computing, and used for a large data volume system based on a traditional database. Background technique [0002] With the development of computer technology, data storage presents a diversity, and the amount of data involved continues to grow rapidly, posing many challenges to data analysis and mining. [0003] In order to process these data, a traditional relational database such as MySQL is used, which processes data through operations such as join, groupby, orderby, and the like. However, the disadvantage of traditional relational databases such as MySQL is that their data processing capabilities are limited. As the amount of data increases, operations such as join, groupby, and orderby appear extremely slow, and even run ou...

AI Technical Summary

Problems solved by technology

This is mainly manifested in ensuring the accuracy of the data, such as processing the conversion of different data types and line breaks, etc., which requires a huge labor cost;

[0007] 2) There is a delay in data processing

The amount of data in the traditional database is too large, and it takes a long time to transfer the data, which cannot meet the needs of real-time business;

[0008] 3) The data of the traditional database needs to be backed up in the distributed system

This is a huge waste of storage space; and

[0009] 4) It is difficult to process across data sources

However, this processing method has a long processing cycle, and there are problems in sqoop's processing of special characters

[0010] Therefore, in the prior art, for businesses that started with traditional databases, when the amount of data increases to a certain extent, the following problems will occur: the data processing speed is too slow, and the data processing process is complicated due to the inconsistency of data source types, slow

Images