Detection system and detection method of advertisement clicking anomaly based on Spark Streaming

Abstract

The invention provides a detection system and a detection method of advertisement clicking anomaly based on Spark Streaming, and relates to the field of computer technique application. Logs are collected when a user clicks the advertisements on a webpage, data collected in real time are cleaned, data field format is standardized, and the standardized data is transferred to the Kafka data information system by Flume, data are classified through a KNN neighborhood algorithm of Spark Streaming, and the three classes of abnormal data, suspicious data, and normal data can be obtained. The abnormal data and the normal data are stored in a database, the suspicious data are sent to the Kafka data information system, and naive Bayes classifiers are trained through the abnormal data, the classification information of the suspicious data can be obtained using the classifier, and data are saved in the database. Advertiser expenses are justly collected by the amount of normal data, in the meantime, the popularities of each advertisement are obtained by analyses, the directions for industrial developments are provided for the advertisers, and the information such as user distributions in the country is provided.

Description

Technical field [0001] The invention relates to the application field of computer technology, in particular to a detection system and a detection method based on Spark Streaming advertisement click abnormality. Background technique [0002] With the explosive growth of data, the era of big data has come. Safe, fast, real-time, and efficient data processing can not only allow enterprises to avoid risks in advance, but also provide timely data and information for enterprise development, product production and development. Effective basis. [0003] However, due to the open nature of the network, it also brings inauthentic information, malicious access, and malicious attacks while making it convenient for the public. This is a problem faced by all open websites. How to prevent these problems, how to extract real and effective data, and reduce the malicious load of the server is the research focus of each open website. Among them, malicious clicks on advertisements are a typical probl...

AI Technical Summary

Problems solved by technology

The current processing technology is generally based on offline batch processing. Such processing technology cannot solve online problems in real time, and cannot quickly provide theoretical basis for some fast decision-making solutions.

For real-time systems such as Storm, although it has the ability to process data in real time, it is weaker than Spark Streaming in terms of data security and large-scale data processing

Images