Real-time log collection and analysis method on basis of B2B (Business to Business) platform

Abstract

The invention discloses a real-time log collection and analysis method on the basis of a B2B (Business to Business) platform. The method comprises the following steps of 1) taking access logs and system logs in the B2B platform as a data source, collecting data of the data source in real time, and storing the data in a register; 2) processing the data through a log parser, and parsing the data according to different formats; 3) collecting log data through a log collector; 4) defining a plurality of subtypes of one type, wherein the subtypes are distributed on nodes of a plurality of servers in a cluster; 5) caching the data of the collector through a distributed memory; 6) processing the data through a distributed calculator; 7) outputting a processing result to a database through the distributed calculator. The real-time log collection and analysis method on the basis of the B2B platform provided by the invention has the advantages that the data are collected in real time and are calculated in parallel in a big data concurrency process, a significant effect in the real-time analysis is achieve, and an obvious advantage in the real-time calculation is obtained.

Description

technical field [0001] The invention relates to a real-time log collection and analysis method based on a B2B platform. Background technique [0002] Since the development of e-commerce, a large number of user visits and a large amount of system log information have been accumulated, including visitors, information providers, etc.; and the browsing actions of such visitors are recorded in the logs, and the system abnormalities and monitoring logs are all Recorded in log files, this type of log is often massive data. [0003] When a user visits our website using a search engine or directly enters a URL through a browser to visit our website, all actions of the user visiting the website will be recorded in the server log file, and it will be recorded which page the user came in from , and the path of the next page will be recorded in the log file of the server, and the log of the user's search on the website will be recorded in the log file. When the user visits the page, if...

AI Technical Summary

Problems solved by technology

[0004] The existing domestic distributed log collection and analysis methods, especially for real-time collection, and in the case of real-time and large data volume, are prone to many problems or cannot guarantee real-time performance, such as :

[0005] Chinese patent CN201310317960.6 provides an implementation plan for a distributed log collection server, which uses a distributed method to complete the collection of massive logs, and collects through multiple collection servers. There is no defined master-slave relationship, and concurrent collection It will cause a log file to be collected by multiple collectors at the same time, which may cause multiple copies of one data, and there is no real-time collection feature

[0006] Chinese patent CN201410061318.0 provides a distributed device log collection method, which uses the intermediary model to build an integrated data middle layer through the distributed log processing framework, forming an integrated data intermediary management service, and the data intermediary service collects device logs Perform distributed storage on each distributed storage point and perform data connection. If you need to increase distributed storage points, use the dynamic expansion mechanism of distributed storage points to realize; build an integrated data middle layer, collect, format and process logs uniformly, Concentrate on the unified management and scheduling of distributed data storage points. In terms of distributed storage, it is proposed to connect with distributed data. In terms of distributed collection and real-time performance, the efficiency is very low. This method is carried out in storage. Improvement, does not involve distributed collection and calculation

Images