Method and system for providing domain name log based on fused CDN manufacturer real-time log

Abstract

The invention relates to a method and system for providing a domain name log based on a fused CDN manufacturer real-time log, and the method comprises the steps: obtaining a real-time log provided by a CDN manufacturer, writing the real-time log into a first distributed log system based on a log receiving service, writing the real-time log in the first distributed log system into a distributed file system, and writing the real-time log in the first distributed log system into a second distributed file system; scheduling a specific calculation engine to segment a real-time log on a distributed file system based on a preset specific workflow platform, generating a segmented log, uploading the segmented log to a specific cloud storage space, generating a log downloading link, and sending the log downloading link to a log demand side. According to the method, distributed system landing and segmentation technologies are combined, so that the log is provided with extremely high expansion capability, the timeliness and the stability of obtaining the log by the user are further greatly improved, and the use requirements of the user are met.

Description

Technical field [0001] The present application relates to the field of log generation, and in particular to a domain name log for a fusion CDN vendor real-time log. Background technique [0002] The full name of the CDN is Content Delivery Network, which is a content distribution network. The CDN is a smart virtual network that is built on the existing network. Relying on the edge server deploying on all localities, the user's load balancing, content distribution, and scheduling function modules, the user needs to obtain the required content, and reduce the network congestion. Improve user access response speed and hit rate. CDN's key technologies mainly include content storage and distribution technology. [0003] Currently, the domain name of the seven cattle fuses CDN is configured on multiple upstream cDNs, so the log provided to the customer comes from multiple upstream CDN vendors, and the currently invigible to the customer, you need to call the domain name configured by m...

AI Technical Summary

Problems solved by technology

[0004] However, the current solution relies on offline logs of manufacturers. Due to the different technical architectures of different CDN manufacturers, the timeliness of providing log download interfaces is not the same. As a result, the logs provided by Qiniu to users are limited by the slowest CDN manufacturer. In addition, in the scheme of actively calling the CDN manufacturer to obtain the download link, it is impossible to know when the CDN manufacturer's download link is ready at the first time, so it can only continuously poll and call the manufacturer's API, resulting in waste of resources; at the same time, the log volume of different domain names varies greatly. The data skew is serious, and concurrency cannot be well utilized. Domain names with a large amount of data can easily block the log processing process. Each domain name is configured in multiple vendors, and each vendor will generate multiple files, so the number of files will also expand many times. Troubled by a large number of small files

These problems have caused great instability to the user log providing system. From time to time, it will be affected by manufacturer fluctuations, network fluctuations, or log system processing capabilities, requiring manual intervention, which in turn affects user use.

Images