Online shop big data processing method and device, equipment and storage medium

A big data processing and data technology, applied in the field of data processing, can solve the problems of partition scan failure, slow execution speed, slow data reading, etc., and achieve the effect of reducing reading failure, scheduling commodity data volume stably, and improving scheduling speed.
CN113672687AActive Publication Date: 2021-11-19北京值得买科技股份有限公司
15 Cites 1 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
北京值得买科技股份有限公司
Publication Date
2021-11-19

Smart Images

  • Figure 1
    Figure 1
  • Figure 2
    Figure 2
  • Figure 3
    Figure 3
Patent Text Reader

Abstract

The invention relates to an online shop big data processing method and device, equipment and a storage medium. The method comprises the following steps: reading commodity data in an Hbase through a Spark program, and generating an elastic distributed data set RDD; converting the RDD into the DataFrame; filtering data in the DataFrame through an online shop website ID and shelve-unshelve conditions; converting the DataFrame with data filtered into the RDD again; matching each piece of data of the reconverted RDD with a pre-configured acquisition scheme and an acquisition task through a MapPartitions operation of the RDD, and generating an optimal scheduling item; and pushing the generated optimal scheduling item into different link pools according to the task frequency. According to the technical scheme provided by the invention, the scheduling speed of the commodity data is greatly improved, the volume of the scheduled commodity data is stable, and the situation of reading failure of partition data is greatly reduced.
Need to check novelty before this filing date? Find Prior Art

Description

technical field

[0001] The present invention relates to the field of data processing, in particular to an e-commerce big data processing method, device, equipment and storage medium. Background technique

[0002] With the accumulation of business data, the amount of e-commerce commodity data is getting larger and larger, and the reading of data is getting slower and slower, resulting in a slowdown of the entire data processing process. Taking the Hbase database as an example, the Hbase database uses partitions as the data unit. There can be many pieces of data in a partition, and 4096 partitions can store up to 200 million commodity data. Use a Python program to traverse each partition sequentially and read commodity data Finally, do scheduling, distribution, download and analysis operations. The problem with this method is: the execution speed is slow, and when scheduling a round of data, it takes 1.5 hours to 2 hours; and, during the execution process, there will be failur...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More