An e-commerce big data processing method, device, equipment and storage medium

Abstract

The invention relates to a method, device, equipment and storage medium for processing big data of e-commerce. The method includes: reading commodity data in Hbase through the Spark program to generate an elastic distributed data set RDD; converting the RDD into a DataFrame; filtering the data in the DataFrame through the ID of the e-commerce website and the off-shelf conditions; DataFrame is re-converted into RDD; through the MapPartitions operation of RDD, each piece of data in the re-converted RDD is matched with the pre-configured collection scheme and collection task to generate the optimal scheduling item; the generated optimal scheduling item is calculated according to the task frequency Pushed into different connection pools. The technical scheme provided by the present invention greatly improves the scheduling speed of commodity data, and the amount of scheduled commodity data is stable, greatly reducing the failure to read partition data.

Description

technical field [0001] The present invention relates to the field of data processing, in particular to an e-commerce big data processing method, device, equipment and storage medium. Background technique [0002] With the accumulation of business data, the amount of e-commerce commodity data is getting larger and larger, and the reading of data is getting slower and slower, resulting in a slowdown of the entire data processing process. Taking the Hbase database as an example, the Hbase database uses partitions as the data unit. There can be many pieces of data in a partition, and 4096 partitions can store up to 200 million commodity data. Use a Python program to traverse each partition sequentially and read commodity data Finally, do scheduling, distribution, download and analysis operations. The problem with this method is: the execution speed is slow, and when scheduling a round of data, it takes 1.5 hours to 2 hours; and, during the execution process, there will be failur...

AI Technical Summary

Problems solved by technology

[0002] With the accumulation of business data, the amount of e-commerce product data is getting larger and larger, and the reading of data is getting slower and slower, resulting in a slowdown of the entire data processing process

Taking the Hbase database as an example, the Hbase database uses partitions as data units. There can be many pieces of data in a partition, and 4096 partitions can store up to 200 million commodity data. Use a Python program to traverse each partition sequentially and read commodity data Finally, do scheduling, distribution, download and analysis operations. The problem with this method is: the execution speed is slow, and when scheduling a round of data, it takes 1.5 hours to 2 hours; and, during the execution process, there will be partition scanning failures , resulting in data that cannot normally enter the downstream process

Images