Log-based user behavior data processing method, medium, device and device

Abstract

The invention discloses a log-based user behavior data processing method, a medium, a device and a device. The method comprises: 1, collecting log information; 2, determining a plurality of access characteristics, and extracting access behavior values of different users for different access characteristics accord to log information; 3, dividing that access characteristic into N groups, and determine the cluster number of each group; N is an integer greater than or equal to 1; step 4, clustering each grouping according to the number of clusters corresponding to the grouping to obtain clusteringresults. The invention solves the problem of poor high-dimensional clustering effect by constructing high-dimensional data as low-dimensional features composed of a plurality of similar or related features and clustering. And by constructing the high-dimensional data into a plurality of similar or related features of the low-dimensional features after clustering, the number of clusters per sub-clustering process is reduced, saving computational time and saving computational memory.

Description

Technical field [0001] The present invention relates to the field of Internet technology, in particular to log-based user behavior data processing methods, media, equipment and devices. Background technique [0002] With the development of Internet services, massive web server (webserver) system logs are generated every day due to user access on the Internet. The web server system log mainly includes: client IP address, client user name, access time, request uri, request status, file size, page link source, client browser and other information. In the prior art, web server system logs are used to classify user behaviors, but the existing classification methods mainly have the following difficulties: [0003] First, when the correspondence between the Uniform Resource Identifier (URI) and its content cannot be known, and the correspondence between the content and the content category cannot be known, it is difficult to use such logs to classify user interests. [0004] Second, the f...

AI Technical Summary

Problems solved by technology

[0003] First, it is difficult to use such logs to classify user interests without knowing the correspondence between Uniform Resource Identifier (URI) and its content, or the correspondence between content and content category

[0004] Second, the feature extraction of the web server is mainly statistical features, such as count, average value, standard deviation, etc., which are access behavior features but do not include access targets, so the access targets of the clustered clusters will be inconsistent, which will lead to Misjudgment

Taking KMeans, the most common clustering method, as an example, the space complexity is O(n*m), where n is the number of data items and m is the number of features. The increase of n and m makes clustering consume a lot of system memory and calculation time increase, while e.g. KMeans is not suitable for generating too many clusters, and dealing with high-dimensional data

This brings challenges to the computational physical consumption of clustering methods and the accuracy of clustering results (such as dissimilar behavior within clusters)

Specifically include: (1) Directly using the general clustering method for high-dimensional data uses a lot of memory and takes a long time

(2) When a large number of access behaviors coexist, the KMeans clustering method needs to predict a large number of clusters. If the number of clusters is estimated to be small, otherwise it will lead to inconsistent behavior within the cluster; if the KMeans method has a large number of clusters, then less effective

Images