Feature engineering recommendation method and device based on spark streaming real-time stream, video website

Abstract

The invention discloses a feature engineering recommendation method based on spark streaming real-time streams. The method comprises the steps that expanded logs and clicked logs of a client are acquired, washed and then added into a distributed message queue; spark streaming is used for subscribing log streams of the expanded logs and the clicked logs, and stream data in the two log streams are combined in engineering; operation is performed on the stream data to generate labels to identify expanded and clicked stream data and expanded and non-clicked stream data; multidimensional features are constructed for the expanded logs and the clicked logs according to basic features, and meanwhile basic time features are combined; the stream data with new features is subjected to off-line training and on-line training to generate recommended stream data. A feature extraction mode widely applicable to most fields is provided, the problem of a small application range of feature engineering is solved, timeliness of feature engineering is realized by adopting the mode of on-line domination and off-line correction, and the effectiveness and accuracy of the features are achieved through a series of feature combination transformation.

Description

Technical field [0001] The present invention relates to the technical field of video recommendation processing, in particular to a feature engineering recommendation method based on spark streaming real-time streaming. Background technique [0002] With the advent of the Internet 2.0 era, the network is flooded with a large amount of information and data. How to use these huge and messy data to dig out valuable information has become a hot topic, which is also an important branch of data mining. The field of machine learning has brought a spring of development. In machine learning technology, few people pay attention to Feature Engineering, but more to the selection and optimization of models and algorithms. However, features are the raw materials of machine learning systems, and their impact on the final model is unquestionable. . [0003] Most models can learn well through the good structure in the data. Even if it is not the best model, high-quality features can also get good ...

AI Technical Summary

Problems solved by technology

[0008] The traditional feature engineering construction relies more on the understanding and experience of the business. After continuous experimental exploration, the features suitable for its own field are obtained, and most of them are offline features, which has strong limitations and lacks timeliness. , which will inevitably lead to the application range of feature engineering is not wide enough, and it is difficult, which is not suitable for platform construction and external promotion

These are not in line with the ecology, platformization, sharing, timeliness, effectiveness and convenience required by today's Internet era, especially cannot meet the needs of video recommendation

Images