Online shop big data processing method and device, equipment and storage medium

Abstract

The invention relates to an online shop big data processing method and device, equipment and a storage medium. The method comprises the following steps: reading commodity data in an Hbase through a Spark program, and generating an elastic distributed data set RDD; converting the RDD into the DataFrame; filtering data in the DataFrame through an online shop website ID and shelve-unshelve conditions; converting the DataFrame with data filtered into the RDD again; matching each piece of data of the reconverted RDD with a pre-configured acquisition scheme and an acquisition task through a MapPartitions operation of the RDD, and generating an optimal scheduling item; and pushing the generated optimal scheduling item into different link pools according to the task frequency. According to the technical scheme provided by the invention, the scheduling speed of the commodity data is greatly improved, the volume of the scheduled commodity data is stable, and the situation of reading failure of partition data is greatly reduced.

Description

technical field [0001] The present invention relates to the field of data processing, in particular to an e-commerce big data processing method, device, equipment and storage medium. Background technique [0002] With the accumulation of business data, the amount of e-commerce commodity data is getting larger and larger, and the reading of data is getting slower and slower, resulting in a slowdown of the entire data processing process. Taking the Hbase database as an example, the Hbase database uses partitions as the data unit. There can be many pieces of data in a partition, and 4096 partitions can store up to 200 million commodity data. Use a Python program to traverse each partition sequentially and read commodity data Finally, do scheduling, distribution, download and analysis operations. The problem with this method is: the execution speed is slow, and when scheduling a round of data, it takes 1.5 hours to 2 hours; and, during the execution process, there will be failur...

AI Technical Summary

Problems solved by technology

[0002] With the accumulation of business data, the amount of e-commerce product data is getting larger and larger, and the reading of data is getting slower and slower, resulting in a slowdown of the entire data processing process

Taking the Hbase database as an example, the Hbase database uses partitions as data units. There can be many pieces of data in a partition, and 4096 partitions can store up to 200 million commodity data. Use a Python program to traverse each partition sequentially and read commodity data Finally, do scheduling, distribution, download and analysis operations. The problem with this method is: the execution speed is slow, and when scheduling a round of data, it takes 1.5 hours to 2 hours; and, during the execution process, there will be partition scanning failures , resulting in data that cannot normally enter the downstream process

Images