Log collection method, device and equipment for content distribution network, medium and product

Abstract

The invention relates to the technical field of computers, and discloses a log collection method and device for a content distribution network, equipment, a medium and a product, and the method comprises the steps that a management server deployed on an edge node obtains a detail log, and the detail log is obtained after processing a log stream transmitted based on a cache server of the edge node; the management server obtains target transmission information, the target transmission information comprises at least one of the network state of the edge node and the disk state of a first disk, the first disk is used for storing detail logs, and the first disk is deployed in the management server; the management server determines a transmission mode of the detail log based on the target transmission information, wherein the transmission mode is active pushing or waiting for pulling; and the management server transmits the detail log to the first central server for storage based on the transmission mode. According to the invention, the problem of long time delay of detail log collection of the content distribution network can be solved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a log collection method, device, equipment, medium and product for a content distribution network. Background Art

[0002] When receiving a user request, the edge node server in the content distribution network will generate a corresponding detailed log based on the user request. The detailed log is used to record information such as the user request of the terminal and the response result of the user request. The detailed log is usually stored on the disk of the cache server of the edge node. If you want to analyze the detailed log of the edge node, you need to pull the detailed log from the cache server of the edge node and store it in the central computer room for subsequent data and log analysis.

[0003] In the related art, detailed logs are mainly pulled from the cache server of the edge node in a timed manner, and the collection delay of the detailed logs is relatively long. Summary of the invention

[0004] In view of this, the present disclosure provides a log collection method, device, equipment, medium and product for a content distribution network to solve the problem of long delay in collecting detailed logs of a content distribution network.

[0005] In a first aspect, the present disclosure provides a log collection method for a content distribution network, which is applicable to a management server deployed on an edge node of the content distribution network, and the method includes:

[0006] Obtaining a detailed log, wherein the detailed log is obtained by processing a log stream transmitted by a cache server of the edge node;

[0007] Obtain target transmission information, where the target transmission information includes at least one of a network status of the edge node and a disk status of a first disk, where the first disk is used to store the detailed log, and the first disk is deployed on the management server;

[0008] Determine a transmission mode of the detailed log based on the target transmission information, wherein the transmission mode is active push or waiting pull;

[0009] The detailed log is transmitted to the first central server for storage based on the transmission method.

[0010] In a second aspect, the present disclosure provides another log collection method for a content distribution network, which is applicable to a central server of the content distribution network, and the method includes:

[0011] Obtain a detailed log generated by a management server of an edge node of a content distribution network, wherein the detailed log is obtained after the management server processes a log stream transmitted by a cache server of the edge node, and a transmission mode of the detailed log is determined by the management server based on target transmission information, wherein the transmission mode is active push or waiting pull, and the target transmission information includes at least one of a network status of the edge node and a disk status of a first disk, wherein the first disk is used to store the detailed log, and the first disk is deployed on the management server;

[0012] The detailed log is stored.

[0013] In a third aspect, the present disclosure provides a log collection device for a content distribution network, which is applicable to a management server deployed on an edge node of the content distribution network, and the device includes:

[0014] A detailed log acquisition module, used to acquire a detailed log, wherein the detailed log is obtained by processing a log stream transmitted by a cache server of the edge node;

[0015] A transmission information acquisition module, used to acquire target transmission information, wherein the target transmission information includes at least one of a network status of the edge node and a disk status of a first disk, wherein the first disk is used to store the detailed log, and the first disk is deployed on the management server;

[0016] A transmission mode determination module, used to determine the transmission mode of the detailed log based on the target transmission information, wherein the transmission mode is active push or waiting pull;

[0017] A detailed log transmission module is used to transmit the detailed log to the first central server for storage based on the transmission method.

[0018] In a fourth aspect, the present disclosure provides another log collection device for a content distribution network, which is applicable to a central server of the content distribution network, and the device includes:

[0019] A detail log collection module, used to obtain a detail log generated by a management server of an edge node of a content distribution network, wherein the detail log is obtained by the management server after processing a log stream transmitted by a cache server of the edge node, and the transmission mode of the detail log is determined by the management server based on target transmission information, wherein the transmission mode is active push or waiting pull, and the target transmission information includes at least one of a network status of the edge node and a disk status of a first disk, wherein the first disk is used to store the detail log, and the first disk is deployed on the management server;

[0020] The detail log storage module is used to store the detail log.

[0021] In a fifth aspect, the present disclosure provides an electronic device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the above-mentioned log collection method of the content distribution network by executing the computer instructions.

[0022] In a sixth aspect, the present disclosure provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the above-mentioned log collection method for a content distribution network.

[0023] In a seventh aspect, the present disclosure provides a computer program product, including computer instructions, where the computer instructions are used to enable a computer to execute the above-mentioned log collection method for a content distribution network.

[0024] The log collection method of the content distribution network provided by the embodiment of the present disclosure processes the storage process of the detailed log through the management server. Since the management server and the cache server of the edge node of the content distribution network are independent of each other, the log stream generated by the cache server does not need to be stored in the cache server. Therefore, it can effectively reduce the disk space and input and output waste of the cache server, improve the disk utilization rate of the cache server, and improve the content distribution efficiency of the content distribution network. At the same time, according to the network status of the edge node and / or the disk status of the first disk, the transmission mode of the detailed log is determined. Therefore, it is possible to adaptively select a suitable transmission mode, and actively push and wait for pull as a backup plan. In a transmission environment where waiting for the detailed log to be pulled will cause a long collection delay, the active push method is used to reduce the collection delay of the detailed log of the content distribution network, thereby improving the collection efficiency of the detailed log. Alternatively, when the amount of data of the detailed log stored in the first disk is too large, the active push method is combined to avoid overflow of the first disk. In addition, it is also possible to combine the waiting and pulling method to avoid actively pushing detailed logs when the amount of data processed by the first central server is too large, thereby avoiding the memory overflow of the first central server, thereby effectively avoiding the defects of the two solutions of active push and waiting and pulling in different transmission environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the specific embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0026] Figure 1is a schematic diagram of an edge node of a content distribution network according to an embodiment of the present disclosure;

[0027] Figure 2 is a flow chart of a first content distribution network log collection method according to an embodiment of the present disclosure;

[0028] Figure 3 is a flow chart of a second content distribution network log collection method according to an embodiment of the present disclosure;

[0029] Figure 4 is a log processing schematic diagram of an edge node according to an embodiment of the present disclosure;

[0030] Figure 5 is a log processing schematic diagram of another edge node according to an embodiment of the present disclosure;

[0031] Figure 6 is a flow chart of a third content distribution network log collection method according to an embodiment of the present disclosure;

[0032] Figure 7 is a flow chart of a fourth content distribution network log collection method according to an embodiment of the present disclosure;

[0033] Figure 8 It is a schematic diagram of processing of actively pushed detailed logs by a central server according to an embodiment of the present disclosure;

[0034] Fig. 9 is a flowchart of a fifth content distribution network log collection method according to an embodiment of the present disclosure;

[0035] Fig.10 It is a schematic diagram of processing of a second index of a detail log by a central cloud according to an embodiment of the present disclosure;

[0036] Fig.11 is a schematic diagram of processing of aggregated logs by a central server according to an embodiment of the present disclosure;

[0037] Fig.12 It is a schematic diagram of the overall process of a log collection method for a content distribution network according to an embodiment of the present disclosure;

[0038] Fig.13 is a structural block diagram of a log collection device for a content distribution network according to an embodiment of the present disclosure;

[0039] Fig.14 is a structural block diagram of another log collection device for a content distribution network according to an embodiment of the present disclosure;

[0040] Fig.15is a structural block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0042] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0043] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0044] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0045] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0046] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0047] Content Delivery Network (CDN) consists of edge node server clusters distributed in different regions, which are used to route user requests to the service nodes closest to the user, thereby reducing user access latency and improving the availability and performance of websites and applications. Content delivery networks can effectively alleviate Internet network congestion and improve the response speed of Internet terminal users accessing websites.

[0048] When receiving a user request, the edge node server in the content distribution network will generate a corresponding detailed log based on the user request. The detailed log is used to record information such as the user request of the terminal and the response result of the user request. The detailed log is usually stored on the disk of the edge node cache server. If you want to analyze the detailed log of the content distribution network or other edge cloud servers, you need to pull the detailed log from the disk of the edge node cache server and store it in the central computer room for subsequent data and log analysis.

[0049] However, this detailed log processing method has the following problems:

[0050] First, the disk capacity and input / output (I / O) capabilities of the edge node cache servers of mainstream content distribution networks are limited, and they are mainly used to cache the content distributed by the content distribution network. Storing detailed logs on the disk of the edge node cache server may cause the disk of the edge node cache server to affect each other, thereby affecting the content distribution network.

[0051] Secondly, the detailed logs of the content distribution network are generally used for billing reconciliation, which places extremely high demands on the integrity of log collection, and needs to be done exactly once (Eaxctly Once). That is, all detailed logs need to be collected to the central computer room exactly once. However, the network conditions of edge nodes are complex and changeable. In some extreme cases, such as edge node network interruption, high network packet loss rate of edge nodes, cross-regional restrictions on edge nodes, etc., the transmission of detailed logs is very likely to be forced to be interrupted. It is necessary to complete the collection of detailed logs as soon as possible after the network is restored, while ensuring the integrity of the collection.

[0052] Third, at present, the mainstream solutions for collecting cloud-edge logs to the central computer room mainly include "push" and "pull". "Push" means that the edge node pushes the detailed log to the central computer room, and "pull" means that the central computer room periodically pulls the detailed log of the edge node according to the metadata of the edge node. Among them, the "push" method has higher real-time performance, but it is easy to crash the central computer room when an emergency occurs, and the capacity problem needs to be solved by limiting the flow and expanding the capacity. The "pull" method will rely on the cloud information platform that maintains the edge node. Therefore, how to improve the real-time and integrity of the detailed log collection of edge nodes as much as possible while avoiding the shortcomings of the two log collection methods of "push" and "pull" has become an urgent problem to be solved.

[0053] In view of this, according to an embodiment of the present disclosure, a log collection method embodiment of a content distribution network is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0054] In this embodiment, a first content distribution network log collection method is provided, which can be used for edge nodes of a content distribution network. Figure 1 As shown, one or more cache servers are deployed on the edge node, such as cache server 1, cache server 2, ..., cache server n, and the cache server is used to cache the content to be distributed by the edge node. At the same time, one or more management servers are also deployed on the edge node, such as management server 1 and management server 2. The management server is used to provide management services, including log services of the content distribution network to which it belongs, and can also include management services of other content distribution networks. Among them, the machines between the management server and the cache server are isolated, and the management server and cache server of the same edge node communicate with each other through the internal network.

[0055] The log collection method of the content distribution network provided in this embodiment is applicable to the management server deployed on the edge node. Figure 2 is a flow chart of a first content distribution network log collection method according to an embodiment of the present disclosure, such as Figure 2 As shown, the process includes the following steps:

[0056] Step S101, obtaining a detailed log, where the detailed log is obtained by processing a log stream transmitted by a cache server based on an edge node.

[0057] Specifically, a log service is deployed on the management server. The log service exposes an interface based on the Transmission Control Protocol (TCP) to the outside. The interface is used to cache the log stream reported by the server.

[0058] Specifically, when the cache server receives an access request from a client, a batch of logs will be generated, and the cache server will stream this batch of logs to the log service of the management server through a long connection of the transmission control protocol. The log service of the management server combines the log stream reported by the cache server (i.e., a batch of logs generated by the cache server) into a batch of logs, copies this batch of logs, and distributes the copied first log to the coroutine of the detail log for processing to obtain the detail log.

[0059] Step S102, obtaining target transmission information, the target transmission information including at least one of a network status of an edge node and a disk status of a first disk, the first disk is used to store detailed logs, and the first disk is deployed on a management server.

[0060] Furthermore, the target transmission information may also include configuration information of the first central server, and the configuration information is used to determine the transmission mode of the detailed log between the management server and the first central server.

[0061] Step S103, determining the transmission mode of the detailed log based on the target transmission information, the transmission mode being active push or waiting for pull.

[0062] It is worth noting that in this embodiment, duplex mode is supported for collecting detailed logs, that is, waiting for pulling and actively pushing are supported for collecting detailed logs. Waiting for pulling means that the management server passively waits for the first central server to pull detailed logs, and actively pushing means that the management server actively pushes detailed logs to the first central server.

[0063] It should be noted that if the target transmission information includes the configuration information of the first central server, then when the network status is normal, the disk status is normal and the configuration information indicates that the current transmission mode is waiting for pull, the transmission mode is determined to be waiting for pull. When the network status is abnormal or the disk status is abnormal and the configuration information indicates that the current transmission mode is active push, the transmission mode is determined to be active push.

[0064] Step S104: transmitting the detailed log to the first central server for storage based on the transmission method.

[0065] Specifically, if the transmission mode is active push, the persistent overwrite transfer state (POST) of the Hypertext Transfer Protocol (HTTP) is used to push the detail log to the first central server. If the transmission mode is waiting for pull, the detail log is written to the first disk, and a corresponding download index (i.e., the second index described below) is generated, and the download index is reported to the first central server. The log service of the management server provides a transmission control protocol interface for the first central server to stream the detail log on the first disk according to the download index.

[0066] It should be noted that the first central server belongs to the server of the central cloud of the content distribution network, and the central cloud is used to manage the first central server. When actively pushing detailed logs, the central cloud will distribute the detailed logs to the corresponding first central server according to business needs. Alternatively, the central cloud distributes the detailed logs to the corresponding first central server according to a preset configuration file. Optionally, a log processing system can be deployed in the central cloud, and the log processing system distributes the detailed logs pushed by the management server.

[0067] The log collection method of the content distribution network provided in this embodiment processes the storage process of the detailed log through the management server. Since the management server and the cache server of the edge node are independent of each other, the log stream generated by the cache server does not need to be stored in the cache server. Therefore, it can effectively reduce the disk space and input and output waste of the cache server, improve the disk utilization of the cache server, and improve the content distribution efficiency of the content distribution network. At the same time, according to the network status of the edge node and / or the disk status of the first disk, the transmission mode of the detailed log is determined. Therefore, it is possible to adaptively select a suitable transmission mode, and actively push and wait for pull as a backup plan. In a transmission environment where waiting for the detailed log to be pulled will cause a long collection delay, the active push method is used to reduce the detailed log collection delay of the content distribution network, thereby improving the collection efficiency of the detailed log. Alternatively, when the amount of data of the detailed log stored in the first disk is too large, the active push method is combined to avoid overflow of the first disk. In addition, it is also possible to combine the waiting and pulling method to avoid actively pushing detailed logs when the amount of data processed by the first central server is too large, thereby avoiding the memory overflow of the first central server, thereby effectively avoiding the defects of the two solutions of active push and waiting and pulling in different transmission environments.

[0068] In this embodiment, a second content distribution network log collection method is provided, which can be used for edge nodes of a content distribution network; wherein a cache server and a management server are deployed on the edge node, the cache server is used to cache the content to be distributed by the edge node, and the management server is used to provide log management services. Specifically, the content distribution network log collection method is applicable to the management server deployed on the edge node, Figure 3 is a flow chart of a second content distribution network log collection method according to an embodiment of the present disclosure, such as Figure 3 As shown, the process includes the following steps:

[0069] Step S201, obtaining a detailed log, which is obtained by processing the log stream transmitted by the cache server of the edge node. Please refer to the above step S101, which will not be described in detail here.

[0070] Step S202, obtaining target transmission information, the target transmission information includes at least one of the network status of the edge node and the disk status of the first disk, the first disk is used to store detailed logs, and the first disk is deployed on the management server. Please refer to the above step S102, which will not be repeated here.

[0071] Step S203, determining the transmission mode of the detailed log based on the target transmission information, the transmission mode being active push or waiting for pull.

[0072] Specifically, the above step S203 includes:

[0073] Step S203a: if the network status of the edge node is abnormal or the disk status of the first disk is abnormal, the transmission mode is determined to be active push.

[0074] It is worth noting that, when the waiting-pull transmission mode is adopted, the data volume of the detail log to be pulled at a time is large, for example, several hundred megabytes. If the network status of the edge node is abnormal, the transmission interruption problem is likely to occur, resulting in a longer delay in the collection of the detail log. In comparison, the active push transmission mode is compared with the waiting-pull transmission mode, and the data volume of the detail log transmitted at a time is smaller, for example, one megabyte or several megabytes. Therefore, the collection delay of the detail log will be relatively small. Therefore, if the network status of the edge node is abnormal, the active push transmission mode is adopted to reduce the collection delay of the detail log. At the same time, since waiting for pulling usually requires the storage of the detail log, if the disk status of the first disk storing the detail log is abnormal, it is likely to cause the transmission error or loss of the detail log. Therefore, if the disk status of the first disk is abnormal, the active push transmission mode is adopted.

[0075] Step S203b: If the network status of the edge node is normal and the disk status of the first disk is normal, the transmission mode is determined to be waiting for pulling.

[0076] It is worth noting that when using the active push transmission method, there may be some emergencies, causing a sudden increase in the data to be processed by the first central server. It is difficult for the first central server to process these data in time, which will cause a large amount of data to accumulate in the memory of the first central server. If the detailed log continues to be pushed to the target server, it is likely to cause the memory of the first central server to overflow, and the first central server will be unavailable. There may even be a chain reaction, which will continue to cause other central servers to be unavailable. Therefore, if the network status of the edge node is normal and the disk status of the first disk is normal, it is determined to adopt the waiting and pulling transmission method, so that when the amount of data to be processed by the first central server is too large or the memory is insufficient, the detailed log is stored on the edge node, and the detailed log is pulled after the amount of data to be processed by the first central server decreases, so that the first central server can process the detailed log more smoothly.

[0077] Furthermore, when the detailed log is transmitted to the first central server for storage by actively pushing the transmission mode, if the actively pushed detailed log transmission fails, the transmission mode of the detailed log is adjusted to wait for pulling.

[0078] Step S204: transmitting the detailed log to the first central server for storage based on the transmission method.

[0079] In some optional implementations, if the transmission mode of the detailed log is active push, the above step S204 includes:

[0080] Step S2041a, generating the first index of the detailed log.

[0081] Step S2042a, pushing the detail log and the first index to the first central server, the first central server is used to store the detail log and record the first index when the first index is different from the index of the historical record.

[0082] Specifically, the above step S2042a includes: compressing the detail log to obtain a compressed detail log; and pushing the compressed detail log and the first index to the first central server.

[0083] Optionally, the detail logs are compressed according to the number and / or size of the detail logs to obtain compressed detail logs.

[0084] Furthermore, the compressed detail log is combined with the first index, and the combined data is pushed to the first central server.

[0085] It is understandable that when the management server pushes the detail log to the first central server, the first central server usually feeds back information to the management server to notify the management server that the detail log has been successfully received / stored. If the delay of the feedback information is long, it is likely that the management server will misjudge that the first central server has not received the detail log, and transmit the detail log to the first central server again, causing the first central server to receive the same detail log multiple times, and it is likely that the same detail log will be stored multiple times. Therefore, the use of the first index can ensure that the first central server only stores the same detail log once.

[0086] In some other optional implementations, if the transmission mode of the detailed log is to wait for pulling, the above step S204 further includes:

[0087] Step S2041b, write the detailed log to the first disk and generate a second index of the detailed log.

[0088] Specifically, the above step S2041b includes: compressing the detail log to obtain a compressed detail log; writing the compressed detail log to the first disk and generating a second index.

[0089] Optionally, the detail logs are compressed according to the number and / or size of the detail logs to obtain compressed detail logs.

[0090] Step S2042b, transmit the second index to the central cloud, the central cloud is used to write the second index into the index message queue, the first central server is determined by preempting the second index in the index message queue, and the first central server is used to pull the detailed log to the first disk based on the second index.

[0091] Specifically, the log service of the management server provides a hypertext transfer protocol interface, and the first central server pulls the detailed log to the first disk in a streaming manner according to the second index and the hypertext transfer protocol interface.

[0092] The log collection method for the content distribution network provided in this embodiment generates a first index of the detail log when the active push method is adopted, and pushes both the detail log and the first index to the first central server. Therefore, it is convenient for the first central server to determine the storage status of the pushed detail log according to the first index, avoid repeated storage of the same detail log, ensure the uniqueness of the detail log collection, and also ensure the integrity of the detail log collection. When the waiting pull method is adopted, the detail log is written to the first disk and a second index is generated, and the second index is sent to the central cloud, which writes the second index into the index message queue, and determines the first central server by preempting the second index. Therefore, the detailed log processing of each central server managed by the central cloud can be balanced, and to a certain extent, a single central server can be prevented from processing a large amount of data at the same time and causing the central server to crash, so as to ensure the stability of the detailed log collection.

[0093] In some optional implementations, the log collection method for a content distribution network of the present disclosure further includes:

[0094] Step a1, obtaining the aggregated log, which is obtained after performing aggregation processing based on the log stream.

[0095] Specifically, the log service of the management server combines the log stream reported by the cache server (i.e., a batch of logs generated by the cache server) into a batch of logs, and copies the batch of logs into a first log and a second log. The first log is used to obtain a detailed log, and the second log is used to obtain an aggregate log. The second log is distributed to the coroutine of the aggregate log for processing to obtain an aggregate log.

[0096] Step a2: Push the aggregated log to the second central server for storage.

[0097] It should be noted that, similar to the push of detailed logs, the central cloud will distribute the aggregated logs to the corresponding second central server according to business needs. Alternatively, the central cloud distributes the aggregated logs to the corresponding second central server according to a preset configuration file. Optionally, a log processing system can be deployed in the central cloud, and the log processing system distributes the aggregated logs pushed by the management server.

[0098] It should be noted that the first central server and the second central server may be the same or different. The central cloud manages one or more central servers, and the first central server and the second central server belong to the one or more central servers managed by the central cloud.

[0099] The log collection method for a content distribution network provided in this embodiment generates detailed logs and aggregated logs for the same log stream for storage. Therefore, the aggregated log can be used to quickly collect the required information from the log stream, thereby improving the efficiency of log analysis. At the same time, the detailed log records the detailed operation information of the edge node and the content distribution network, and can provide complex log analysis processing. Thus, the advantages of the aggregated log and the detailed log can be combined to meet different log analysis requirements.

[0100] In some optional implementations, the above step a1 includes:

[0101] Step a11, formatting the log stream based on the preset attribute field to obtain a formatted log.

[0102] Optionally, the preset attribute fields include timestamp, domain name and log data. It should be noted that the above preset attribute fields are only examples and can be adjusted according to actual conditions, such as attribute fields such as log type.

[0103] Exemplarily, taking the preset attribute fields including timestamp time_stamp, domain name domain and log data bytes as an example, after the log stream is formatted, a formatted log in the form of “{"time_stamp":xxx,"domain":xxx,"bytes":xxx}” can be obtained.

[0104] Step a12, performing aggregation processing on the formatted log based on the field value of the target attribute field to obtain an aggregated log, and the target attribute field belongs to the preset attribute field.

[0105] Optionally, the target attribute fields include a timestamp and a domain name.

[0106] Specifically, the formatted logs are windowed and aggregated based on the timestamp and domain name to obtain aggregated logs, wherein the aggregated logs include aggregated logs of different time units, for example, aggregated logs at the second level and aggregated logs at the minute level.

[0107] The log collection method for a content distribution network provided in this embodiment formats the log stream based on a preset attribute field to obtain a formatted log. The formatted log is then aggregated based on the field value of the target attribute field to obtain an aggregated log. Therefore, the logs can be classified and collected by the target attribute field to facilitate subsequent log analysis and processing, thereby improving the efficiency of log analysis.

[0108] In some optional implementations, the log collection method for a content distribution network of the present disclosure further includes:

[0109] Step b1, recording the transmission status of the target log, where the target log includes at least one of an aggregate log and a detail log that is actively pushed.

[0110] Specifically, the detailed log and the aggregated log are pushed in the form of a hypertext transfer protocol request, and the log service of the edge node records the transmission status of each hypertext transfer protocol request initiated to the first central server and the second central server.

[0111] Step b2: if the transmission status of the target log is abnormal, the abnormal target log is written to the second disk and waits for the next push. The second disk is deployed on the management server.

[0112] Specifically, if the first and second central servers are abnormal due to various reasons, the log server of the edge node can report the target log of the transmission abnormality, and the management server will persist the abnormal target log to the second disk and wait for the next push.

[0113] It should be noted that the first disk and the second disk belong to the same disk, or may belong to different disks of the management server.

[0114] In the log collection method for a content distribution network provided in this embodiment, if the transmission status of the target log is abnormal, the abnormal target log is written to the second disk and waits for the next push. Therefore, the integrity and consistency of the detailed log and aggregated log collection can be guaranteed.

[0115] As a specific example, Figure 4 and Figure 5As shown, in the edge node, if the cache server receives an access request from the client, a batch of logs is generated, and this batch of logs is streamed to the log service of the management server in the form of a log stream. The log service of the management server monitors the TCP port. If the log stream transmitted by the cache server is monitored, the log stream is copied, and the copied logs are respectively distributed to the coroutines corresponding to the detail log and the aggregate log. In the coroutine of the detail log, the copied first log is processed to obtain the detail log. Determine whether the detail log is actively pushed. If so, an HTTP request is initiated to the first central server based on this batch of detail logs. The edge nodes / dedicated lines of other edge clouds push this batch of detail logs to the first central server for storage by using cloud-edge acceleration or reporting to the public network. If not, the detailed log is split and compressed according to the dynamic configuration, the compressed detailed log is written to the first disk and the corresponding second index is generated, an HTTP request is initiated to the first central server based on the second index, and the second index is transmitted to the central cloud by the edge node / dedicated line of other edge clouds through cloud-edge acceleration or reporting to the public network. The central cloud writes the second index to the index message queue, and the central server managed by the central cloud preempts the second index to determine the first central server. The first central server pulls the detailed log to the first disk for storage according to the second index.

[0116] In the log aggregation process, the copied second log is formatted to obtain a formatted log, and the formatted log is aggregated to obtain a second-level aggregated log and / or a minute-level aggregated log. An HTTP request is initiated based on the aggregated log, and the second index is transmitted to the second central server for storage by edge nodes / dedicated lines of other edge clouds through cloud-edge acceleration or reporting to the public network.

[0117] If the push of the detail log or the aggregate log fails, the abnormal detail log or the aggregate log is reported, and the management server stores the abnormal detail log or the aggregate log in the second disk and waits for the next push.

[0118] In this embodiment, a third log collection method for a content distribution network is provided, which is applicable to a central server managed by a central cloud of the content distribution network. Figure 6 is a flow chart of a third content distribution network log collection method according to an embodiment of the present disclosure, such as Figure 6 As shown, the process includes the following steps:

[0119] Step S301, obtain the detailed log generated by the management server of the edge node of the content distribution network, the detailed log is obtained after the management server processes the log stream transmitted by the cache server of the edge node, the transmission mode of the detailed log is determined by the management server based on the target transmission information, the transmission mode is active push or waiting pull, the target transmission information includes at least one of the network status of the edge node and the disk status of the first disk, the first disk is used to store the detailed log, and the first disk is deployed on the management server.

[0120] It should be noted that the detailed log generation method can refer to the log collection method of the content distribution network applicable to the management server, which will not be described in detail here.

[0121] Specifically, a detailed log receiving service and a detailed log pulling service are deployed on the central server managed by the central cloud. The two log services are used to obtain detailed logs generated by the management server of the edge node of the content distribution network respectively.

[0122] Step S302, storing the detailed log.

[0123] Specifically, the detailed log is stored in the message queue of the detailed log, wherein the message queue of the detailed log is the log distributed publish-subscribe message system (Kafka) middleware of the central server.

[0124] The log collection method of the content distribution network provided in this embodiment processes the storage process of the detailed log through the management server. Since the management server and the cache server of the edge node are independent of each other, the log stream generated by the cache server does not need to be stored in the cache server. Therefore, it can effectively reduce the disk space and input and output waste of the cache server, improve the disk utilization of the cache server, and improve the content distribution efficiency of the content distribution network. At the same time, according to the network status of the edge node and / or the disk status of the first disk, the transmission mode of the detailed log is determined. Therefore, it is possible to adaptively select a suitable transmission mode, and actively push and wait for pull as a backup plan. In a transmission environment where waiting for the detailed log to be pulled will cause a long collection delay, the active push method is used to reduce the detailed log collection delay of the content distribution network, thereby improving the collection efficiency of the detailed log. Alternatively, when the amount of data of the detailed log stored in the first disk is too large, the active push method is combined to avoid overflow of the first disk. In addition, it is also possible to combine the waiting and pulling method to avoid actively pushing detailed logs when the amount of data processed by the first central server is too large, thereby avoiding the memory overflow of the first central server, thereby effectively avoiding the defects of the two solutions of active push and waiting and pulling in different transmission environments.

[0125] This embodiment provides a fourth log collection method for a content distribution network, which is applicable to a central server managed by a central cloud of the content distribution network. Figure 7 is a flow chart of a fourth content distribution network log collection method according to an embodiment of the present disclosure, such as Figure 7 As shown, the process includes the following steps:

[0126] Step S401, obtain the detailed log generated by the management server of the edge node of the content distribution network, the detailed log is obtained after the management server processes the log stream transmitted by the cache server of the edge node, the transmission mode of the detailed log is determined by the management server based on the target transmission information, the transmission mode is active push or waiting pull, the target transmission information includes at least one of the network status of the edge node and the disk status of the first disk, the first disk is used to store the detailed log, and the first disk is deployed on the management server. Please refer to the above step S301, which will not be repeated here.

[0127] Step S402, storing the detailed log.

[0128] Specifically, if the transmission method of the detailed log is active push, the above step S402 includes:

[0129] S4021, obtaining a first index of the detail log and an index of the historical record, wherein the first index is generated by the management server when the transmission mode is active push.

[0130] Specifically, the detailed log and the first index are pushed to the central server in the form of an HTTP request, and the central server obtains the detailed log and the first index by parsing the HTTP request.

[0131] It should be noted that the method for generating the first index may refer to the log collection method of the content distribution network applicable to the management server, which will not be described in detail here.

[0132] S4022: If the first index is different from the index of the historical record, the detailed log is stored and the first index is recorded.

[0133] Specifically, the log service that actively pushes the detailed log records each first index transmitted by the edge node, and if the first index does not conflict with the index of the historical record, the detailed log corresponding to the first index is stored.

[0134] It should be noted that if the first index is consistent with the index of the historical record, the detailed log is not stored.

[0135] In the log collection method for a content distribution network provided in this embodiment, if the first index is different from the index of the historical record, the detailed log is stored and the first index is recorded. Therefore, the first index can be used to ensure that the first central server stores the same detailed log only once, thereby ensuring the uniqueness of the detailed log collection.

[0136] As a specific example, Figure 8 As shown, when the transmission mode of the detail log is active push, the management server of the edge node packages the detail log and the first index and generates an HTTP request. The detail log receiving service of the central server receives the HTTP request of the edge node, parses the HTTP request, and obtains the detail log and the first index. And determine whether the first index conflicts with the index of the historical record. If there is a conflict, return a successful report to the management server. If there is no conflict, the detail log is delivered to the message queue of the detail log for storage. Determine whether the detail log is delivered successfully. If so, return a successful delivery. If not, return a delivery failure.

[0137] This embodiment provides a fifth log collection method for a content distribution network, which is applicable to a central server managed by a central cloud of the content distribution network. Fig. 9 is a flow chart of a fifth content distribution network log collection method according to an embodiment of the present disclosure, such as Fig. 9 As shown, the process includes the following steps:

[0138] Step S501, obtain the detailed log generated by the management server of the edge node of the content distribution network, the detailed log is obtained after the management server processes the log stream transmitted by the cache server of the edge node, the transmission mode of the detailed log is determined by the management server based on the target transmission information, the transmission mode is active push or waiting pull, the target transmission information includes at least one of the network status of the edge node and the disk status of the first disk, the first disk is used to store the detailed log, and the first disk is deployed on the management server.

[0139] Specifically, if the transmission method of the detail log is to wait for pulling, the above step S501 includes:

[0140] S5011, preempt the second index of the detail log in the index message queue. The second index is generated when the management server writes the detail log to the first disk when the transmission mode is waiting for pull. The second index in the index message queue is written by the central cloud after receiving the second index transmitted by the management server.

[0141] It should be noted that the method for generating the second index may refer to the log collection method of the content distribution network applicable to the management server, which will not be described in detail here.

[0142] S5012: If the second index is seized, the detailed log is pulled to the first disk based on the second index.

[0143] Specifically, the download link corresponding to the second index is accessed, and the detailed log is pulled to the first disk through the download link.

[0144] It should be noted that if the second index is not preempted, the second index of the next batch of detail logs will be preempted.

[0145] Step S502, storing the detailed log, which can be referred to as the above step S302, will not be described in detail here.

[0146] In the log collection method of the content distribution network provided in this embodiment, when the central cloud receives the second index of the detailed log, it writes the second index into the index message queue, and the central server managed by the central cloud processes the detailed log by preempting the second index. Therefore, the detailed log processing of each central server managed by the central cloud can be balanced, and to a certain extent, a single central server can be prevented from crashing due to processing a large amount of data at the same time, so as to ensure the stability of detailed log collection.

[0147] As a specific example, Fig.10 As shown, when the transmission mode of the detail log is waiting to be pulled, the management server of the edge node generates an HTTP request based on the second index of the detail log, and the central cloud receives the HTTP request of the edge node, parses the HTTP request, and obtains the second index. The second index is written into the index message queue to determine whether the second index is written successfully. If so, a report of success is returned to the management server. If not, a report of failure is returned to the management server. The central server managed by the central cloud preempts the second index in the index message queue through the detail log pulling service, and pulls the detail log corresponding to the second index to the first disk of the management server for storage.

[0148] It can be understood that the log collection method of the content distribution network disclosed in the present invention adopts a batch processing idea and a two-stage download method for pulling detailed logs. First, the management server of the edge node generates a corresponding second index for the detailed log, and reports the second index to the central cloud through an HTTP request. After the log service of the central cloud that pulls the detailed log receives the second index, it writes the second index to the index message queue, and different central servers preempt the second index in the index message queue. The central server that preempts the second index pulls the detailed log from the first disk by accessing the download link corresponding to the second index. Then, the pulled detailed log is written into the message queue of the detailed log. This two-stage download method can effectively ensure the integrity and real-time nature of detailed log collection.

[0149] In some optional implementations, the log collection method for a content distribution network of the present disclosure further includes:

[0150] Step c1, in the process of pulling the detail log, record the data information of the first data that has been pulled in the detail log.

[0151] Optionally, the data information is the number of log lines.

[0152] It should be noted that when the central server pulls the detail logs, it needs to ensure that all generated detail logs are pulled to the central server as much as possible. Therefore, a strict batch processing solution can be adopted. At the same time, in order to ensure that the detail logs are not pulled repeatedly, each batch of detail logs needs to be micro-batch processed according to the number of log lines, and the number of log lines that have been streamed to the central server is recorded.

[0153] Step c2: if an exception occurs in the process of pulling the detail log, the second data in the detail log is re-pulled based on the data information, and the second data includes the data in the detail log except the first data.

[0154] Specifically, when the collection is retried due to an exception, the remaining uncollected data in the detail log is collected according to the number of log lines that have been recorded, so as to implement the collection with the semantics of "Exactly Once".

[0155] The log collection method for a content distribution network provided in this embodiment records the data information of the first data that has been pulled in the detailed log during the process of pulling the detailed log, and if an exception occurs in the pulling process of the detailed log, the remaining data that has not been pulled in the detailed log is re-pulled based on the data information. Therefore, the uniqueness and integrity of the detailed log can be ensured.

[0156] In some optional implementations, the log collection method for a content distribution network of the present disclosure further includes: receiving an aggregate log transmitted by a management server, where the aggregate log is obtained by the management server performing aggregation processing on the log stream; and storing the aggregate log.

[0157] Specifically, an aggregate log receiving service is deployed on the central cloud. The aggregate log receiving service dynamically delivers aggregate logs of different types and dimensions to different aggregate log message queues. An offline database and a real-time database are set downstream of the aggregate log message queue to consume and store the aggregate logs respectively.

[0158] It should be noted that the method for generating the aggregate log may refer to the log collection method of the content distribution network applicable to the management server, which will not be described in detail here.

[0159] The log collection method for a content distribution network provided in this embodiment generates detailed logs and aggregated logs for the same log stream for storage. Therefore, the aggregated log can be used to quickly collect the required information from the log stream, thereby improving the efficiency of log analysis. At the same time, the detailed log records the detailed operation information of the edge node and the content distribution network, and can provide complex log analysis processing. Thus, the advantages of the aggregated log and the detailed log can be combined to meet different log analysis requirements.

[0160] In some optional implementations, the aggregate logs are stored in an offline database and an online database. The log collection method for a content distribution network disclosed herein further includes: performing data processing on the aggregate logs in the offline database to obtain processed aggregate logs; and using the processed aggregate logs to overwrite the corresponding aggregate logs in the online database.

[0161] Understandably, since the stored aggregate logs may contain some inaccurate data, it is necessary to process these data using offline calculations in the offline database to correct them. In this process, the aggregate logs stored in the offline database will deviate from the aggregate logs stored in the online database. Therefore, it is necessary to use the processed aggregate logs to overwrite the corresponding aggregate logs in the online database to ensure the consistency of aggregate log storage.

[0162] The log collection method for a content distribution network provided in this embodiment processes the aggregate logs in the offline database, thereby ensuring the accuracy of the stored aggregate logs. The processed aggregate logs are used to overwrite the corresponding aggregate logs in the online database. Therefore, the consistency of the aggregate logs stored in the offline database and the online database can be ensured.

[0163] As a specific example, Fig.11 As shown, the management server of the edge node pushes the aggregated log to the central server, and the aggregated log receiving service of the central server receives the transmitted aggregated log and stores the aggregated log in the offline database and the online database respectively. The aggregated log in the offline database is processed and the processed aggregated log is used to overwrite the corresponding aggregated log in the online database.

[0164] As a specific example, Fig.12As shown, the process of the entire log collection service is as follows: if the cache server receives an access request from the client, a batch of logs is generated, and the batch of logs is streamed to the log service of the management server in the form of a log stream. The log service of the management server copies the log stream and distributes the copied logs to the coroutines corresponding to the detail log and the aggregate log respectively. In the coroutine of the detail log, the copied first log is processed to obtain the detail log. It is determined whether the detail log is actively pushed. If so, based on pushing the detail log to the central server managed by the central cloud, the detail log receiving service of the central server writes the detail log to the message queue of the detail log. If not, the detail log is compressed and stored on the first disk and the corresponding second index is generated, and the second index is pushed to the central cloud. The central cloud writes the second index to the index message queue. The central server managed by the central cloud preempts the second index, and the detail log pulling service of the central server directly pulls the detail log to the first disk according to the preempted second index, or pulls the detail log through other edge node agents. The detail log pulling service writes the detail log to the message queue of the detail log. Furthermore, the detailed logs in the message queue of the detailed logs are written into the full log database, or log hot spot analysis is performed based on the detailed logs.

[0165] In the log aggregation process, the copied second log is formatted and aggregated to obtain the aggregate log. The management server pushes the aggregate log to the central server, and the aggregate log receiving service of the central server writes the aggregate log to the message queue of the aggregate log. The offline database and the real-time database aggregate and process the aggregate log based on the batch-stream integration method, and store the processed data in the monitoring and billing database.

[0166] As a specific application example, the edge nodes of the content distribution network are deployed with a cache server and a management server. The management server obtains the detailed logs and the aggregated logs through the log collection method of the content distribution network applicable to the management server disclosed in the present invention, and transmits the detailed logs and the aggregated logs to the central server managed by the central cloud. The central server collects and stores the detailed logs and the aggregated logs of the edge nodes through the log collection method of the content distribution network applicable to the central server disclosed in the present invention, so that the central server can perform data and log analysis based on the stored detailed logs and aggregated logs.

[0167] It is worth noting that in the related art, there are some open source log collection components developed based on the User Datagram Protocol (UDP). This type of open source log collection component is difficult to guarantee the reliability of log collection and log transmission. In contrast, the log collection method of the content distribution network disclosed in the present invention uses the TCP protocol as the base from log production to cloud-edge transmission, which can effectively guarantee the reliability and consistency of log collection and log transmission. At the same time, the log collection method of the content distribution network disclosed in the present invention stores the logs in the management server of the edge node, which solves the risk of the mainstream edge node logs and distribution content reusing the same disk.

[0168] Moreover, the log collection method of the content distribution network disclosed in the present invention proposes a "duplex mode" transmission method for detailed logs, which can improve the real-time and integrity of the log collection of the edge cloud as much as possible while avoiding the shortcomings of the two log collection methods of "push" and "pull", and provide double protection for the log collection performance and integrity. Moreover, the log collection method of the content distribution network disclosed in the present invention adopts the idea of ​​micro-batch processing for the big data processing of detailed logs from the edge nodes of the edge cloud to the central servers of the central cloud, which can have better log collection performance and scalability while ensuring good integrity.

[0169] In addition, the logs of the content distribution network are generally used for billing and reconciliation, and there are extremely high requirements for the integrity of log collection, which needs to be executed only once. That is, all logs (including detailed logs and aggregated logs) need to be completely collected to the central server. It has been proved through actual measurements that the log collection method of the content distribution network disclosed in the present invention can achieve a log integrity of 99% in 10 minutes and 99.99% in 60 minutes, and the integrity of log collection is relatively good.

[0170] It should be noted that the log collection method for a content distribution network disclosed in the present invention is not only applicable to content distribution networks, but also to all edge cloud services.

[0171] In this embodiment, a log collection device for a content distribution network is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated hereafter. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0172] This embodiment provides a log collection device for a content distribution network, which is applicable to a management server deployed on an edge node of the content distribution network, such as Fig.13 As shown, including:

[0173] A detailed log acquisition module 601 is used to acquire detailed logs, which are obtained by processing the log stream transmitted by the cache server of the edge node;

[0174] The transmission information acquisition module 602 is used to acquire target transmission information, where the target transmission information includes at least one of a network status of an edge node and a disk status of a first disk, where the first disk is used to store detailed logs and is deployed on a management server;

[0175] The transmission mode determination module 603 is used to determine the transmission mode of the detailed log based on the target transmission information, and the transmission mode is active push or waiting pull;

[0176] The detail log transmission module 604 is used to transmit the detail log to the first central server for storage based on the transmission method.

[0177] In some optional implementations, the transmission mode determination module 603 includes:

[0178] A first determining unit, configured to determine that the transmission mode is active push if the network status of the edge node is abnormal or the disk status of the first disk is abnormal;

[0179] The second determining unit is configured to determine that the transmission mode is waiting for pulling if the network status of the edge node is normal and the disk status of the first disk is normal.

[0180] In some optional implementations, the detailed log transmission module 604 includes:

[0181] A first index generating unit, configured to generate a first index of the detail log if the transmission mode of the detail log is active push;

[0182] The detail log transmission unit is used to push the detail log and the first index to the first central server. The first central server is used to store the detail log and record the first index when the first index is different from the index of the historical record.

[0183] In some optional implementations, the detailed log transmission module 604 further includes:

[0184] A second index generating unit, configured to write the detail log to the first disk and generate a second index of the detail log if the transmission mode of the detail log is to wait for pulling;

[0185] The index transmission unit is used to transmit the second index to the central cloud, and the central cloud is used to write the second index into the index message queue. The first central server is determined by preempting the second index in the index message queue, and the first central server is used to pull the detailed log to the first disk based on the second index.

[0186] In some optional implementations, the log collection device of the content distribution network of the present disclosure further includes:

[0187] Aggregate log acquisition module, used to obtain aggregate logs, which are obtained after aggregating the log stream;

[0188] The aggregate log transmission module is used to push the aggregate log to the second central server for storage.

[0189] In some optional implementations, the aggregate log acquisition module includes:

[0190] A formatting unit, used to format the log stream based on preset attribute fields to obtain a formatted log;

[0191] The aggregation processing unit is used to perform aggregation processing on the formatted log based on the field value of the target attribute field to obtain an aggregated log, and the target attribute field belongs to a preset attribute field.

[0192] In some optional implementations, the log collection device of the content distribution network of the present disclosure further includes:

[0193] A status recording module, used to record the transmission status of a target log, where the target log includes at least one of an aggregate log and a detail log that is actively pushed;

[0194] The abnormal log writing module is used to write the abnormal target log to the second disk and wait for the next push if the transmission status of the target log is abnormal. The second disk is deployed on the management server;

[0195] The abnormal log retransmission module is used to push the abnormal target log in the second disk to the third central server again.

[0196] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0197] The log collection device of the content distribution network in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0198] In this embodiment, another log collection device for a content distribution network is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated hereafter. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0199] This embodiment provides another log collection device for a content distribution network, which is applicable to a central server of a content distribution network, such as Fig.14 As shown, including:

[0200] The detailed log collection module 701 is used to obtain the detailed log generated by the management server of the edge node of the content distribution network. The detailed log is obtained after the management server processes the log stream transmitted by the cache server of the edge node. The transmission mode of the detailed log is determined by the management server based on the target transmission information. The transmission mode is active push or waiting pull. The target transmission information includes at least one of the network status of the edge node and the disk status of the first disk. The first disk is used to store the detailed log, and the first disk is deployed on the management server.

[0201] The detail log storage module 702 is used to store the detail log.

[0202] In some optional implementations, the detailed log collection module 701 includes:

[0203] An index preemption unit, used for preempting the second index of the detail log in the index message queue if the transmission method of the detail log is waiting for pulling, the second index is generated by the management server when the detail log is written to the first disk when the transmission mode is waiting for pulling, and the second index in the index message queue is written by the central cloud after receiving the second index transmitted by the management server;

[0204] The log pulling unit is used to pull the detailed log to the first disk based on the second index if the second index is preempted.

[0205] In some optional implementations, the log collection device of the content distribution network of the present disclosure further includes:

[0206] An information recording module, used for recording data information of the first data that has been pulled in the detail log during the process of pulling the detail log;

[0207] The re-pulling module is used to re-pull the second data in the detail log based on the data information if an exception occurs in the pulling process of the detail log, and the second data includes the data in the detail log except the first data.

[0208] In some optional implementations, the log collection device of the content distribution network of the present disclosure further includes:

[0209] The aggregate log collection module is used to receive the aggregate log transmitted by the management server. The aggregate log is obtained by the management server through the aggregation processing of the log stream;

[0210] The aggregate log storage module is used to store aggregate logs.

[0211] In some optional implementations, the aggregate log is stored in an offline database and an online database. The log collection device of the content distribution network of the present disclosure also includes:

[0212] The data processing module is used to process the aggregate logs in the offline database to obtain the processed aggregate logs;

[0213] The log overwriting module is used to overwrite the corresponding aggregate log in the online database with the processed aggregate log.

[0214] In some optional implementations, the detailed log storage module 702 includes:

[0215] An index acquisition unit, used to acquire a first index of the detail log and an index of the historical record, wherein the first index is generated by the management server when the transmission mode is active push;

[0216] The detail log storage unit is used to store the detail log and record the first index if the first index is different from the index of the historical record.

[0217] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0218] The log collection device of the content distribution network in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0219] The present disclosure also provides an electronic device having the above Fig.13 or Fig.14 The log collection device of the content distribution network shown.

[0220] See also Fig.15 , Fig.15 is a structural block diagram of an electronic device provided by an optional embodiment of the present disclosure, such as Fig.15As shown, the electronic device includes: one or more processors 801, memory 802, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can store instructions executed in the electronic device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig.15 A processor 801 is taken as an example.

[0221] The processor 801 may be a central processing unit, a network processor or a combination thereof. The processor 801 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable logic gate array, a general purpose array logic or any combination thereof.

[0222] The memory 802 stores instructions executable by at least one processor 801, so that the at least one processor 801 executes the method shown in the above embodiment.

[0223] The memory 802 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 802 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 802 may optionally include a memory remotely arranged relative to the processor 801, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0224] The memory 802 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 802 may also include a combination of the above types of memory.

[0225] The electronic device also includes a communication interface 803, which is used for the electronic device to communicate with other devices or a communication network.

[0226] The embodiments of the present disclosure also provide a computer-readable storage medium. The above-mentioned method according to the embodiments of the present disclosure can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium and downloaded through a network, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0227] A part of the present disclosure may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present disclosure through the operation of the computer. Those skilled in the art should understand that the existence of computer program instructions in computer-readable media includes, but is not limited to, source files, executable files, installation package files, etc., and accordingly, the way in which computer program instructions are executed by a computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to the computer.

[0228] Although the embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A log collection method for a content distribution network, characterized in that: Applicable to a management server deployed on an edge node of a content distribution network, the method comprising: Obtaining a detailed log, wherein the detailed log is obtained by processing a log stream transmitted by a cache server of the edge node; Acquire target transmission information, where the target transmission information includes at least one of a network status of the edge node and a disk status of a first disk, where the first disk is used to store the detailed log, and the first disk is deployed on the management server; Determine a transmission mode of the detailed log based on the target transmission information, wherein the transmission mode is active push or waiting pull; The detailed log is transmitted to the first central server for storage based on the transmission method.

2. The log collection method of the content distribution network according to claim 1, characterized in that: The determining the transmission mode of the detailed log based on the target transmission information includes: If the network status of the edge node is abnormal or the disk status of the first disk is abnormal, determining that the transmission mode is active push; If the network status of the edge node is normal and the disk status of the first disk is normal, the transmission mode is determined to be waiting for pulling.

3. The log collection method of a content distribution network according to claim 1, characterized in that: If the transmission mode of the detailed log is active push, then transmitting the detailed log to the first central server for storage based on the transmission mode includes: Generating a first index of the detailed log; The detail log and the first index are pushed to the first central server, and the first central server is used to store the detail log and record the first index when the first index is different from the index of the historical record.

4. The log collection method of a content distribution network according to claim 1, characterized in that: If the transmission mode of the detailed log is to wait for pulling, then transmitting the detailed log to the first central server for storage based on the transmission mode includes: Writing the detailed log to the first disk and generating a second index of the detailed log; The second index is transmitted to the central cloud, and the central cloud is used to write the second index into the index message queue. The first central server is determined by preempting the second index in the index message queue, and the first central server is used to pull the detailed log to the first disk based on the second index.

5. The log collection method of a content distribution network according to claim 1, characterized in that: The method further comprises: Obtaining an aggregated log, where the aggregated log is obtained after performing aggregation processing on the log stream; The aggregated log is pushed to the second central server for storage.

6. The log collection method of a content distribution network according to claim 5, characterized in that: The obtaining of the aggregate logs includes: Formatting the log stream based on a preset attribute field to obtain a formatted log; The formatted log is aggregated based on the field value of the target attribute field to obtain the aggregated log, and the target attribute field belongs to the preset attribute field.

7. The log collection method for a content distribution network according to any one of claims 1 to 6, characterized in that: The method further comprises: Recording the transmission status of a target log, wherein the target log includes at least one of an aggregate log and the detail log actively pushed; If the transmission status of the target log is abnormal, the abnormal target log is written to the second disk and waits for the next push, and the second disk is deployed on the management server.

8. A log collection method for a content distribution network, characterized in that: Applicable to a central server of a content distribution network, the method comprises: Obtain a detailed log generated by a management server of an edge node of a content distribution network, wherein the detailed log is obtained after the management server processes a log stream transmitted by a cache server of the edge node, and a transmission mode of the detailed log is determined by the management server based on target transmission information, wherein the transmission mode is active push or waiting pull, and the target transmission information includes at least one of a network status of the edge node and a disk status of a first disk, wherein the first disk is used to store the detailed log, and the first disk is deployed on the management server; The detailed log is stored.

9. The log collection method of a content distribution network according to claim 8, characterized in that: If the transmission method of the detailed log is active push, the storing of the detailed log includes: Obtaining a first index of the detail log and an index of the historical record, wherein the first index is generated by the management server when the transmission mode is active push; If the first index is different from the index of the historical record, the detailed log is stored and the first index is recorded.

10. The log collection method of a content distribution network according to claim 8, characterized in that: If the transmission method of the detailed log is to wait for pulling, then obtaining the detailed log generated by the management server of the edge node of the content distribution network includes: Preempting the second index of the detail log in the index message queue, the second index is generated by the management server writing the detail log to the first disk when the transmission mode is waiting for pulling, and the second index in the index message queue is written by the central cloud after receiving the second index transmitted by the management server; If the second index is seized, the detail log is pulled to the first disk based on the second index.

11. The log collection method of a content distribution network according to claim 10, characterized in that: The method further comprises: In the process of pulling the detailed log, recording data information of the first data that has been pulled in the detailed log; If an exception occurs in the process of pulling the detail log, the second data in the detail log is re-pulled based on the data information, where the second data includes data in the detail log other than the first data.

12. The log collection method of a content distribution network according to claim 8, characterized in that: The method further comprises: receiving an aggregated log transmitted by the management server, where the aggregated log is obtained by the management server performing aggregation processing on the log stream; The aggregation log is stored.

13. The log collection method of a content distribution network according to claim 12, characterized in that: The aggregate log is stored in an offline database and an online database, and the method further includes: Performing data processing on the aggregate log in the offline database to obtain a processed aggregate log; The processed aggregate log is used to overwrite the corresponding aggregate log in the online database.

14. A log collection device for a content distribution network, characterized in that: A management server deployed on an edge node of a content distribution network, the device comprising: A detailed log acquisition module, used to acquire a detailed log, wherein the detailed log is obtained by processing a log stream transmitted by a cache server of the edge node; A transmission information acquisition module, used to acquire target transmission information, wherein the target transmission information includes at least one of a network status of the edge node and a disk status of a first disk, wherein the first disk is used to store the detailed log, and the first disk is deployed on the management server; A transmission mode determination module, used to determine the transmission mode of the detailed log based on the target transmission information, wherein the transmission mode is active push or waiting pull; A detailed log transmission module is used to transmit the detailed log to the first central server for storage based on the transmission method.

15. A log collection device for a content distribution network, characterized in that: A central server applicable to a content distribution network, the device comprising: A detail log collection module, used to obtain a detail log generated by a management server of an edge node of a content distribution network, wherein the detail log is obtained by the management server after processing a log stream transmitted by a cache server of the edge node, and the transmission mode of the detail log is determined by the management server based on target transmission information, wherein the transmission mode is active push or waiting pull, and the target transmission information includes at least one of a network status of the edge node and a disk status of a first disk, wherein the first disk is used to store the detail log, and the first disk is deployed on the management server; The detail log storage module is used to store the detail log.

16. An electronic device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the log collection method for a content distribution network according to any one of claims 1 to 7, or executes the log collection method for a content distribution network according to any one of claims 8 to 13 by executing the computer instructions.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the log collection method for a content distribution network according to any one of claims 1 to 7, or to execute the log collection method for a content distribution network according to any one of claims 8 to 13.

18. A computer program product, characterized in that The method comprises computer instructions, wherein the computer instructions are used to enable a computer to execute the log collection method for a content distribution network according to any one of claims 1 to 7, or to execute the log collection method for a content distribution network according to any one of claims 8 to 13.

Images