Method and device for realizing ID mapping based on Spark framework
A framework and -ID technology, applied in the field of big data, can solve problems such as blind programmatic transactions, inability to achieve real-time bidding and precise delivery, and inability to quickly and accurately obtain mapping results for massive user data
Pending Publication Date: 2020-09-22
BEIJING QIHOO TECH CO LTD
View PDF0 Cites 1 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
Without ID Mapping, programmatic transactions will be blind, unable to achieve real-time bidding and precise placement
[0004] At present, the main technical bottleneck of ID Mapping lies in the inability to quickly and accurately obtain mapping results when processing massive user data
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0112] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided for more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.
[0113] At present, ID Mapping cannot effectively extract the ID relationship network when faced with a large amount of user data, a large number of ID types and numbers, and complex relationships between IDs, making it difficult to effectively implement in engineering.
[0114] The Spark framework is a fast and general-purpose cluster computing platform designed for large-scale data processing. It enables memory distribution data se...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention provides a method and a device for realizing ID mapping based on a Spark framework. The method mainly comprises the steps of performing data preprocessing on a two-dimensional ID relation table to obtain an initial number-ID pair relation table; splitting and aggregating the initial number-ID pair relation table by taking ID as a key to obtain a plurality of initial number one-time aggregation subsets; taking the initial number as a key, and splitting and aggregating the plurality of initial number primary aggregation subsets to obtain an initial number aggregation subset result;numbering the initial number aggregation subset result by using a unified identifier to obtain a unified identifier-initial number aggregation subset relationship table; and according to the unifiedidentifier-initial number aggregation subset relationship table and the initial number-ID pair relationship table, obtaining a corresponding relationship between the unified identifiers and the IDs, thereby realizing unified representation of the IDs. According to the method, operations such as storage, filtering, splitting and aggregation of massive user data sets are realized, and the efficiency, accuracy and reliability of an ID mapping algorithm are improved.
Description
technical field [0001] The invention relates to the technical field of big data, in particular to a method and device for realizing ID mapping based on a Spark framework, a computer storage medium and a computing device. Background technique [0002] ID Mapping is a basic and critical technology in the field of big data. Simply put, ID-Mapping is to identify several pieces of data from different sources as the same user or subject through some technical means. For example, a certain user Zhang San uses AA Mobile Assistant on the first mobile phone, uses Baidu Maps on the second mobile phone, watches iQiyi videos on the tablet computer, and uses the AA browser on the personal computer. A mobile phone, a second mobile phone and a tablet often share the same wifi, and the second mobile phone is often connected to a PC through a data cable, so how do you determine this based on the behavior of the objects on these 4 devices and the connections between them? The four objects ar...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/2458G06Q30/02
CPCG06Q30/0251G06Q30/0275G06F16/2471
Inventor 赵林马征王斌峰李晓明
Owner BEIJING QIHOO TECH CO LTD