Page access upstream and downstream flow calculation method
A calculation method and page access technology, applied in the field of big data analysis, can solve problems such as the inability to reflect the gradual decrease of traffic, repeated statistics, and the inability to meet the analysis needs of traffic conditions, and achieve the effect of speeding up query matching efficiency and facilitating implementation.
Pending Publication Date: 2021-04-06
GUANGZHOU FAISCO INFORMATON TECH
View PDF4 Cites 0 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
The current problem is that the solution for direct query and calculation of access logs is complex and cumbersome, and many analysis services on the market can only meet basic data requirements, and cannot well match the company's business situation and analysis needs, especially market analysis. The service calculates the upstream and downstream traffic algorithm. When there are multiple matches in the user's access path, all of them will be counted. There is a problem of repeated statistics, which cannot reflect the data index of the traffic decreasing layer by layer, and cannot match the traffic of a given page access path. Situation analysis needs
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment 1
[0031] Example 1, such as figure 1 As shown in , a method for calculating upstream and downstream traffic of page visits includes the following steps:
[0032] Obtain user access logs from the Nginx reverse proxy server, transmit and upload them to HDFS storage, use the Hive offline processing engine to analyze the access logs, and extract the logs including: access time, user ID, session ID, current page link and source Page links are stored in the Hive data table, and the storage format uses Parquet or ORC to improve storage query efficiency.
[0033] Next, when processing the access log, the Hive offline processing engine is also used for processing, and the access path rule is implemented by encapsulating the logic of the UDAF function, combined with the HQL grouping based on the user ID, and the access path of the same user is input to the UDAF function for processing. After processing, the output is serialized binary data of a special data structure, which is stored as ...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention belongs to the technical field of big data analysis, and relates to a page access upstream and downstream flow calculation method, which comprises the following steps of: obtaining access log data of a user and storing the access log data; enabling the offline analysis system to read and analyze the access log data, restoring the user access path tree, and writing the user access path tree into a data table in the storage system; and enabling the ad hoc analysis system to read and match the access tree of the user according to the query condition, and returning the upstream and downstream traffic data of the page. According to the method, the structure-free and out-of-order access logs are converted into the tree structure of the complete access path of the user through the access path calculation rule, so that various query matching conditions are supported, and repeated matching of the data is avoided more accurately; and through a storage mode of serializing a tree structure, under the guarantee that access information is not lost, the query matching efficiency is accelerated by the link tree dictionary unit block, the serializing mode of the access tree single block element is greatly convenient for realizing a path retrieval algorithm, and the capability of providing a given access path and matching can be provided.
Description
technical field [0001] The invention belongs to the technical field of big data analysis, and relates to a method for calculating upstream and downstream traffic of page access. Background technique [0002] Through the access logs left by users visiting the website, analyzing page access and traffic conversion to downstream pages are the basic operational indicators of each company. Based on these data, page layout and other means can be adjusted to maximize user access retention and adjust traffic distribute. The current problem is that the solution for direct query and calculation of access logs is complex and cumbersome, and many analysis services on the market can only meet basic data requirements, and cannot well match the company's business situation and analysis needs, especially market analysis. The service calculates the upstream and downstream traffic algorithm. When there are multiple matches in the user's access path, all of them will be counted. There is a pro...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/958G06F16/955G06F16/951
CPCG06F16/972G06F16/9558G06F16/951
Inventor 刘家锹
Owner GUANGZHOU FAISCO INFORMATON TECH