Element processing method and device, electronic equipment and storage medium

Abstract

The invention discloses an element processing method and device, equipment and a storage medium. The method comprises the steps of: receiving an element as a target object; writing the target object into a target bucket, the target bucket being a bucket matched with the target object in a plurality of preset buckets, each bucket having a container, the bucket being used for storing elements havingthe same behavior, and the container being used for storing feature values of the elements; in the target bucket, calculating the cardinal number of the characteristic value in the container; and identifying the attribute of the target object according to the cardinal number. According to the embodiment of the invention, the attribute identification of the target object can be realized under a hundred million-level data scale, the calculation efficiency can be improved, and the memory occupation is reduced.

Description

technical field [0001] Embodiments of the present invention relate to big data processing technologies, and in particular, to an element processing method, device, electronic equipment, and storage medium. Background technique [0002] With the development of the mobile Internet, the amount of global data is growing astonishingly, and the data and software and hardware infrastructures such as mobile devices and IP (Internet Protocol) are also very large, up to tens of billions. Huge scale. [0003] Faced with such a huge data scale, all walks of life on the mobile Internet often require to quickly distinguish new and old data information from massive data in a very short period of time in the processing of business data. In the fields of advertising promotion, news push, product sales, live video and other fields, it is often necessary to quickly count the current number of views and clicks, and to retain the information of new users. For example, in the mobile Internet adv...

AI Technical Summary

Problems solved by technology

[0004]The b tree is to maintain a data structure that can compare the size of elements, and to realize whether an element exists in a certain set through efficient search and insertion of elements Accurate judgment, the disadvantage is that as the number of elements increases, the memory occupied by the b-tree also increases rapidly, and the upper limit of the memory is uncontrollable. In the Internet environment where the data scale continues to explode and the single-machine memory is limited, there is a risk of memory overflow. The stable operation of the system is affected, and the b-tree cannot be used to quickly combine multiple sets

Bitmap realizes high-speed search of elements by mapping each element to a bit with a hash function, because any two bitmaps can be merged as long as the bitwise OR operation is performed, so it can solve the problem that the b-tree cannot quickly find the union The problem, but the disadvantage is that memory needs to be pre-allocated, and the upper limit of memory determines the upper limit of the cardinality that can be obtained, and when the cardinality is close to this limit, the judgment accuracy will drop sharply due to hash value conflicts

Images