Content delivery network log management system
By designing a content distribution network log management system, using the combination of access end, storage end and task scheduling modules, the problem of large amount of CDN log data and different user needs is solved, and efficient and flexible log processing and query capabilities are achieved.
Patent Information
- Application Number
- CN202411746709.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-29
- Publication Date
- 2025-04-15
AI Technical Summary
How to design a high-performance content distribution network (CDN) offline log processing system that is flexible to respond to user needs and is suitable for the problem of huge amount of CDN log data and different user needs.
Provided is a content distribution network log management system, including access-end components, storage-end components and task scheduling modules. The access terminal component receives and parses the initial log data, generates data keys, and forms index log data. The storage component receives and stores the index log data, and merges and transmits data under the request of the task scheduling module. The task scheduling module receives user review requests, generates data merging requests, and distributes them to storage components for processing.
By generating data keys and forming index log data, the efficiency of log data query and scheduling is improved. Through data transmission and merging of storage components, the log data required by users is sorted out, and the log files expected by users are generated quickly and efficiently. The design of the task scheduling module allows the system to flexibly meet the different data review needs of different users, and to deal with different data magnitude and traffic surges by expanding the access and storage components.
Smart Images
Figure CN119829537A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical field of content distribution networks, and in particular to a content distribution network log management system. Background Art
[0002] Content Delivery Network (CDN) is an excellent network architecture that can improve the speed at which users access Internet content. By deploying a large number of server nodes in various regions, CDN can cache content on servers closer to users, thereby reducing the distance that content must be transmitted and improving access speed and response time.
[0003] When accessing images, text or video resources with the help of Content Delivery Network (CDN), CDN is prone to generate a large amount of user access log data. CDN manufacturers usually need to collect and process these log data in real time. After processing, they need to generate offline logs in the user data format according to user requirements for users to download, so as to facilitate subsequent analysis.
[0004] However, since the amount of CDN log data is huge and the log generation requirements of each user are generally different, how to design a high-performance offline log processing system that can flexibly respond to user needs is still a huge challenge for relevant personnel. Summary of the invention
[0005] In view of this, one or more embodiments of the present disclosure provide a content distribution network log management system, which can efficiently manage content distribution network logs and flexibly meet users' log viewing needs.
[0006] The present disclosure provides a content distribution network log management system, the system comprising: an access end module including at least one access end component, a storage end module including at least one storage end component, and a task scheduling module; wherein the access end component is used to receive and parse initial log data; generate a data key corresponding to the initial log data according to the parsed information of the initial log data; mark the initial log data by the data key to form index log data; send the index log data to the storage end component; the storage end component is used to receive and store the index log data; in response to a data merge request sent by the task scheduling module, transmit and merge the index log data between the storage end components; the task scheduling module is used to receive a user review request; generate at least one data merge request according to the user review request; send the data merge request to the storage end component, so that the storage end component transmits and merges the index log data according to the data merge request.
[0007] The content distribution network log management system provided by the present disclosure can generate data keys through access-end components, mark initial log data, and form index log data, which is conducive to improving the efficiency of log data query and scheduling; through storage-end components, index log data can be transmitted and merged, and log data required by users can be sorted out, which is convenient for quickly and efficiently generating log files expected by users; through task scheduling modules, data merging requests can be disassembled from user access requests, and data merging requests can be distributed to storage-end components for processing, so as to flexibly meet different data access requirements of different users.
[0008] In addition, both the access-end components and the storage-end components can be expanded, ensuring the scalability of the management system. This is conducive to dealing with log data of different data volumes and can also effectively cope with traffic surges.
[0009] In one embodiment, the parsing of the initial log data includes: obtaining configuration information of a log database of the initial log data; parsing the initial log data according to the configuration information to determine domain name information of the initial log data; judging whether to process the initial log data in shards according to the configuration information; and if the initial log data is processed in shards, generating a shard value for the initial log data.
[0010] Determine the domain name information of the initial log data so that the data key generated based on the parsed information can identify log data from different domain names. Sharding the initial log data can effectively deal with data explosion. Based on the generated shard value, the data key can accurately identify the sharded log data.
[0011] In one embodiment, sending the index log data to the storage-end component includes: obtaining a storage service list; determining a candidate storage component in the storage-end component according to the storage service list; and sending the index log data to the candidate storage component.
[0012] In the content distribution network log management system provided by the present disclosure, storage end components can be flexibly added and deleted. The storage service list can accurately indicate the currently available storage end components, and index log data can be sent to these currently available storage end components for data storage and processing.
[0013] In one embodiment, the task scheduling module includes multiple task scheduling nodes, and the task scheduling nodes are divided into a master scheduling node and at least one slave scheduling node through a master selection mechanism; wherein the master scheduling node is used to receive the user review request, generate the data merge request, and send the data merge request, and the slave scheduling node is used as a redundant guarantee for the master scheduling node.
[0014] The design of the master scheduling node and the slave scheduling node ensures the reliability of the task scheduling module. The slave scheduling node can provide data backup function, and when the master scheduling node fails, it can use the master election mechanism to quickly elect a new master scheduling node, thereby ensuring the continuity and high availability of the log management service.
[0015] In one embodiment, generating at least one data merging request based on the user's query request includes: determining at preset time intervals whether there is a data query request included in the user's query request in the current time segment; if the data query request exists in the current time segment, responding to the data query request and generating at least one data merging request.
[0016] Each user can have customized requirements for querying log data or generating log files. The preset time granularity is used to determine whether there is a data query request included in the user's query request, which can flexibly meet the customized data query requirements of different customers.
[0017] In one embodiment, generating at least one data merge request includes: traversing the storage-end component, obtaining the data key of the index log data and the component identifier of the storage-end component, and recording the matching relationship between the data key and the component identifier; placing the data key into a first memory queue to form a full set key; placing the component identifier into a second memory queue; for the first identifier in the second memory queue, extracting a first subset key from the full set key, and generating a first merge request, wherein the first merge request is used to indicate the first storage component corresponding to the first identifier, and merge the index log data corresponding to the first subset key.
[0018] The traversal operation can determine all currently available storage components and the index log data saved by these storage components to avoid omissions. Through two memory queues, data merge requests can be allocated to different storage components in an orderly manner to avoid omissions and duplications. Each storage component can process a part of the data merge task according to the first merge request. Multiple storage components coordinate and process in a distributed manner to jointly complete the data merge task of the management system, thereby improving data processing efficiency.
[0019] In one embodiment, generating at least one data merge request includes: if the first storage component corresponding to the first identifier has processed the data merge task indicated by the first merge request, then putting the first identifier back into the second memory queue; determining whether there are remaining keys among the full set of keys in the first memory queue; if there are remaining keys among the full set of keys, then for the second identifier in the second memory queue, extracting a second subset of keys from the remaining keys, and generating a second merge request, wherein the second merge request is used to instruct the second storage component corresponding to the second identifier to merge the index log data corresponding to the second subset of keys.
[0020] For data merging tasks with small data volumes, the corresponding storage components can complete the tasks quickly and re-enter the second memory queue to wait for reuse. For data merging tasks with large data volumes, the corresponding storage components can continue to maintain the data processing logic without being disturbed. In this way, the data skew problem can be effectively solved.
[0021] In one embodiment, in response to the data merge request sent by the task scheduling module, the index log data is transmitted and merged between the storage-end components, including: in response to the data merge request, the third storage component in the storage-end component determines the target key indicated by the data merge request and the target storage component that matches the target key; the third storage component communicates with the target storage component to obtain the target log data corresponding to the target key in the target storage component; the third storage component merges the target log data.
[0022] The target log data corresponding to the target key is uniformly received from the target storage component. Since the individual files belong to the same batch, data can be sent in batches to improve system performance.
[0023] In one embodiment, obtaining the target log data corresponding to the target key in the target storage component includes: identifying the target domain name contained in the target key; establishing a Bloom filter for each target domain name; obtaining the target log data and calculating the target hash value of the target log data; placing the target hash value into the corresponding Bloom filter according to the domain name information of the target log data to determine whether the target log data already exists; if the target log data already exists, generating reversal data for the target log data, and the reversal data is used for billing.
[0024] By establishing a Bloom filter, you can quickly query and filter large amounts of log data to determine whether the log data is repeated. For repeated target log data, you can generate correction data to avoid repeated billing of the target log data.
[0025] In one embodiment, in response to the data merge request sent by the task scheduling module, the index log data is transmitted and merged between the storage-end components, including: checking whether the data merge request has been processed; if the data merge request has been processed, returning; if the data merge request has not been processed, checking whether the data merge request is being processed; if the data merge request is being processed, returning; otherwise, starting the transmission and merging of the index log data.
[0026] Since there may be node takeover in the task scheduling module, the same data merge request may be sent repeatedly. After receiving the data merge request, first determine whether it has been processed or is being processed, which can avoid repeated execution of the data merge request, avoid resource waste and concurrent failures. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] The features and advantages of the embodiments of the present disclosure may be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present disclosure in any way. In the accompanying drawings:
[0028] Figure 1 A schematic diagram of functional modules of a content distribution network log management system in one embodiment of the present disclosure is shown;
[0029] Figure 2 A schematic diagram of the organizational structure of a content distribution network log management system in one embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0030] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.
[0031] In the related art, CDN manufacturers need to collect and process CDN logs in real time, and need to generate offline logs in the user data format for customers to download according to customer requirements. However, each customer can have a different number of domain names. Customers usually require CDN manufacturers to generate offline download logs for each domain name in a customer-defined format. At the same time, each customer can also have other personalized customization requirements. For example, each customer needs to generate log files in different time spans. Some customers need to generate log files once an hour, and some customers need to generate log files once every fifteen minutes or even five minutes. For another example, each customer needs different log file subpackaging rules. Some customers require a single log file to be less than 500M, some customers require all log data to be written to the same log file, and some customers require a single log file to be less than 1G. For another example, each customer needs different naming rules for log files, and some customers require log files to have consecutive sequence numbers.
[0032] In the related technologies, the data size of CDN logs is relatively large. In addition, during the operation of the business, CDN traffic may surge in a short period of time, which requires the log management system to be able to meet the instantaneous traffic situation.
[0033] In view of this, a content distribution network log management system provided in an embodiment of the present disclosure can flexibly respond to the log viewing needs of customers and is a stable and high-performance log processing system.
[0034] See also Figure 1 The content distribution network log management system provided by one embodiment of the present disclosure may include an access end module 100, a storage end module 200 and a task scheduling module 300. The access end module may include at least one access end component. The storage end module may include at least one storage end component.
[0035] In this embodiment, the access end component can be used to receive and parse the initial log data; generate a data key corresponding to the initial log data based on the parsed information of the initial log data; mark the initial log data through the data key to form index log data; and send the index log data to the storage end component.
[0036] In this embodiment, the storage-end component can be used to receive and store the index log data; in response to the data merging request sent by the task scheduling module, the index log data is transmitted and merged between the storage-end components.
[0037] In this embodiment, the task scheduling module can be used to receive user requests; generate at least one data merge request based on the user request; and send the data merge request to the storage-end component so that the storage-end component transmits and merges the index log data based on the data merge request.
[0038] In this embodiment, the initial log data can be a CDN log collected by a CDN edge machine. The access-end component can generate a data key, mark the initial log data, and form index log data, which is conducive to improving the efficiency of log data query and scheduling. The storage-end component can transmit and merge index log data, sort out the log data required by the user, and facilitate the rapid and efficient generation of the user's desired log file. The task scheduling module can disassemble the data merge request from the user's access request, and can distribute the data merge request to the storage-end component for processing, so as to flexibly meet the different data access requirements of different users.
[0039] In this implementation, both the access-end component and the storage-end component can be expanded, which ensures the scalability of the content distribution network log management system, is conducive to dealing with log data of different data levels, and can also effectively deal with traffic surges.
[0040] In some embodiments, the parsing of the initial log data includes: obtaining configuration information of a log database of the initial log data; parsing the initial log data according to the configuration information to determine domain name information of the initial log data; judging whether to process the initial log data in shards according to the configuration information; and if the initial log data is processed in shards, generating a shard value for the initial log data.
[0041] Specifically, the configuration information of the log database can be dynamically maintained. By obtaining the latest configuration information of the log database, the initial log data can be accurately parsed. According to the configuration information, the received initial log data can be parsed to a certain extent to parse out the necessary calculation dimensions. For example, the domain name can be parsed from the initial log data, the customer ID can be calculated based on the domain name, and the hash value of the entire data can be calculated.
[0042] Determine the domain name information of the initial log data so that the data key generated based on the parsed information can identify log data from different domain names. Sharding the initial log data can effectively deal with data explosion. Based on the generated shard value, the data key can accurately identify the sharded log data.
[0043] In some implementations, sending the index log data to the storage-end component includes: obtaining a storage service list; determining a candidate storage component in the storage-end component based on the storage service list; and sending the index log data to the candidate storage component.
[0044] Specifically, in the content distribution network log management system provided by the present disclosure, storage end components can be flexibly added and deleted. By using the storage service list, the currently available storage end components can be accurately indicated (considering that some storage end components are faulty or not enabled), and the index log data can be sent to these currently available storage end components for data storage and processing.
[0045] In some embodiments, the task scheduling module includes multiple task scheduling nodes, and the task scheduling nodes are divided into a master scheduling node and at least one slave scheduling node through a master selection mechanism; wherein the master scheduling node is used to receive the user review request, generate the data merge request, and send the data merge request, and the slave scheduling node is used as a redundant guarantee for the master scheduling node.
[0046] The design of the master scheduling node and the slave scheduling node ensures the reliability of the task scheduling module. The slave scheduling node can provide data backup function, and when the master scheduling node fails, it can use the master election mechanism to quickly elect a new master scheduling node, thereby ensuring the continuity and high availability of the log management service.
[0047] In some embodiments, generating at least one data merge request based on the user query request includes: determining at preset time granularity whether there is a data query request included in the user query request in the current time segment; if the data query request exists in the current time segment, responding to the data query request and generating at least one data merge request.
[0048] Specifically, each user may have customized requirements for querying log data or generating log files. By presetting the time granularity at intervals, it is determined whether there is a data query request included in the user's query request, which can flexibly meet the customized data query requirements of different customers. The preset time granularity can be the minimum time granularity set based on experience, such as half a minute, one minute, five minutes, etc., to ensure that the actual application requirements are met.
[0049] In some embodiments, generating at least one data merge request includes: traversing the storage-end components, obtaining the data key of the index log data and the component identifier of the storage-end component, and recording the matching relationship between the data key and the component identifier; placing the data key into a first memory queue to form a full set key; placing the component identifier into a second memory queue; for the first identifier in the second memory queue, extracting a first subset key from the full set key, and generating a first merge request, wherein the first merge request is used to indicate the first storage component corresponding to the first identifier, and merge the index log data corresponding to the first subset key.
[0050] Specifically, the traversal operation can determine all currently available storage components (taking into account faulty components and unenabled components), as well as the index log data saved by these storage components, to avoid omissions. Through two memory queues, it can be ensured that data merge requests are orderly assigned to different storage components for execution, avoiding omissions and duplications. Each storage component can process a part of the data merge task based on the first merge request. Multiple storage components are distributed and coordinated to jointly complete the data merge task of the management system, improving data processing efficiency.
[0051] In some embodiments, generating at least one data merge request includes: if the first storage component corresponding to the first identifier has processed the data merge task indicated by the first merge request, then putting the first identifier back into the second memory queue; determining whether there are remaining keys for the full set of keys in the first memory queue; if there are remaining keys for the full set of keys, then for the second identifier in the second memory queue, extracting a second subset of keys from the remaining keys, and generating a second merge request, wherein the second merge request is used to instruct the second storage component corresponding to the second identifier to merge the index log data corresponding to the second subset of keys.
[0052] Specifically, for data merging tasks with small data volumes, the merging can be completed quickly, and the corresponding storage components can re-enter the second memory queue and be reused to perform new data merging tasks. For data merging tasks with large data volumes, the corresponding storage components can continue to maintain the data processing logic without being disturbed. In this way, the data skew problem can be effectively solved.
[0053] In some embodiments, in response to the data merge request sent by the task scheduling module, the index log data is transmitted and merged between the storage-end components, including: in response to the data merge request, the third storage component in the storage-end component determines the target key indicated by the data merge request, and the target storage component that matches the target key; the third storage component communicates with the target storage component to obtain the target log data corresponding to the target key in the target storage component; the third storage component merges the target log data.
[0054] Specifically, the third storage component can uniformly receive the target log data corresponding to the target key from the target storage component. Because the individual files all belong to the same batch, data can be sent in batches to improve system performance.
[0055] In some embodiments, obtaining the target log data corresponding to the target key in the target storage component includes: identifying the target domain name contained in the target key; establishing a Bloom filter for each target domain name; obtaining the target log data and calculating the target hash value of the target log data; placing the target hash value into the corresponding Bloom filter according to the domain name information of the target log data to determine whether the target log data already exists; if the target log data already exists, generating reversal data for the target log data, and the reversal data is used for billing.
[0056] Specifically, a Bloom filter is established to quickly query and filter large-scale log data to determine whether the log data is repeated. A Bloom filter is a highly space-efficient probabilistic data structure that can be used to determine whether an element is in a set. It allows some false positives but no false negatives.
[0057] In the related art, since log data may fail to be sent during transmission, there may be a resending mechanism for the log data, which may cause the log data to be charged repeatedly, resulting in inaccurate billing.
[0058] In contrast, the technical solution provided by the present disclosure can generate correction data for repeated target log data. The correction data can correct the repeated data, ensure the accuracy of the target log data, and avoid repeated billing of the target log data.
[0059] In some embodiments, in response to the data merge request sent by the task scheduling module, the index log data is transmitted and merged between the storage-end components, including: checking whether the data merge request has been processed; if the data merge request has been processed, returning; if the data merge request has not been processed, checking whether the data merge request is being processed; if the data merge request is being processed, returning; otherwise, starting the transmission and merging of the index log data.
[0060] Specifically, due to the possibility of node takeover in the task scheduling module, the same data merge request may be sent repeatedly. After receiving the data merge request, first determine whether it has been processed or is being processed, which can avoid repeated execution of the data merge request, avoid resource waste and concurrent failures.
[0061] See also Figure 2 , a content distribution network log management system provided by an embodiment of the present disclosure is mainly divided into three components: a data access terminal, a data storage terminal and a task scheduling module.
[0062] After CDN log data is collected from CDN edge machines by collectors, it can be sent to a WebServer (web server) in a certain (several) designated central computer room for centralized processing via the Hypertext Transfer Protocol (HTTP). These log data will be stored on disk or in memory for a period of time.
[0063] In this embodiment, the data access terminal may be a component of the central computer room, which is mainly responsible for processing the log data at the WebServer and then sending it to the data storage terminal. The data storage terminal may store the log data sent by the data access terminal and perform subsequent processing.
[0064] In this embodiment, the access-end component in the data access end can be responsible for parsing the received log data to a certain extent, and after parsing out the necessary calculation dimensions (such as parsing the domain name from the log data, calculating the customer ID based on the domain name, calculating the hash value of the entire log data, etc.), it is sent to the storage-end component of the data storage end.
[0065] In this embodiment, the storage component of the data storage end can receive the log data sent by the access end component, and store the data on the disk according to certain data organization principles, and respond to the data reading request of the task scheduling system and send the log data to the task scheduling module.
[0066] In this implementation, the task scheduling module can read the corresponding configuration, generate corresponding scheduling tasks, and perform log data merging processing according to the download requirements of different customers.
[0067] The component structure of the data access end and the data storage end ensures the high availability and horizontal expansion of the system.
[0068] In this implementation, some open source servers for coordination services (such as ZooKeeper, ZK for short) may be used to manage configuration information, name data, provide distributed synchronization, provide group services, and the like.
[0069] In a practical application example, a content distribution network log management system is deployed based on ZK. The details of the system are as follows.
[0070] 1. Access terminal functions:
[0071] Monitor the database configuration and reload it when it changes.
[0072] Monitor the zk directory / storage-server, obtain the list of backend storage-servers (storage nodes), and monitor changes in storage-servers.
[0073] Receive log data sent by the edge host. According to the database configuration, obtain the format of the log data, parse the domain name, and determine whether the domain name needs to be sharded according to the database configuration. The shards here can be dynamically adjusted from the database to respond to the data surge in time. If there is a data surge, the number of shards can be adjusted according to the actual situation.
[0074] Set the key of the log data, and randomly select a storage-server from the list of storage-servers to send the log data with the key set. The key information can include the domain name and shard value.
[0075] 2. Storage function:
[0076] Receive the data sent by the access end and write the data to the specified location according to the key. The key can contain not only information such as customer ID and domain name, but also the customer's time requirements. For example, if the customer requires data to be uploaded every 15 minutes, then a file will be written for each 15-minute time period.
[0077] Respond to the scheduling system's request to send data and send the data with the specified key to the requester.
[0078] It should be noted that when a storage component on the storage side is started, a temporary storage-server can be registered on the zk / storage-server path.
[0079] 3. Scheduling module:
[0080] First, when the scheduling module is started, it can register with zk, start multiple instances, use zk to select a master, and select a master (master node) to generate offline data merging tasks on a scheduled basis.
[0081] Then, the scheduling module can start the timer, which can be in minutes. Since the customer's offline data generation time limit is generally more than half an hour, a 1-minute granularity can usually meet customer needs.
[0082] When the timer is running, it first determines whether there is a customer who needs to generate data in this time slice (for example, if a customer requires a log file to be generated every 15 minutes, then at the time points of 0, 15, 30, and 45, it can be determined that there is a customer who needs to generate data).
[0083] If a customer needs to generate data, all storage-servers are traversed, and all data keys are required to be sent to the scheduling master, the corresponding relationship between the key and the host is saved, and the key is placed in the first memory queue. At the same time, the master obtains the host identifiers of all storage-servers and places them in the second memory queue. For each host identifier in the second memory queue, a sub-item is taken from the full set of the key, a MergeDataRequest (data merge request) is generated, and sent to the corresponding storage-server to perform the data merge logic. When a storage-server completes the MergeDataRequest, the master host can put the host identifier of this storage-server back into the second memory queue. If there is still unprocessed data in the key, the merge request for the next key is sent to this storage-server. This can effectively handle the data skew problem. Small data can be merged quickly, and storage-servers can be reused quickly. For large data, the data processing logic can continue. Repeat this process until the key data is completely processed.
[0084] When new customers are added or the amount of customer-level data increases dramatically, the system capacity can be expanded horizontally by continuously adding new storage-servers. The processing capacity of the entire system is limited only by the amount of data of a single key. When a single domain name of a certain customer increases dramatically, the number of keys can be dynamically increased by increasing the number of shards. Sharding the data volume of a single domain name to a reasonable amount of data can effectively reduce the data volume of a single key.
[0085] 4. Data merging logic: After a single storage-server receives MergeDataRequest, it can check whether the request has been processed normally by other storage-servers (check the zk path, this situation will occur after other nodes take over after the master fails), or check whether the request is being processed by other storage-servers. If so, return. If the request has not been processed, you can get the list of machines with this key from the request, then establish connections with these machines, and ask these machines to send this part of the data to themselves, and at the same time establish a Bloom filter for each customer or domain name. At this time, batch data can be sent, because individual files belong to the same batch, and can be sent over the network through functions similar to sendfile (file call), improving system performance. When the merge is in progress, you can register a temporary node with a domain name + time period in zk, and write the storage-server id to indicate that the merge is in progress (you can use the scheduling-job tag). After the merge is successful, you can register a permanent node with a domain name + time period in zk to indicate that the process has been successful (you can use the scheduled-job tag), and delete the temporary node above.
[0086] 5. Data deduplication: After receiving new data, the storage-server calculates the hash value of each piece of data and puts it into the corresponding Bloom filter to determine whether the data exists. If it does, it generates the corresponding billing data reversal data and puts it into the event stream platform (such as Kafka) for billing calculation. There is a certain error here, and it is proposed to improve the accuracy of the Bloom filter to reduce the data misjudgment rate. Since only a small number of users or domain names have a large amount of data, improving the accuracy of the Bloom filter will not increase too much resource usage.
[0087] 6. System horizontal scalability: When the number of edge machines increases, the number of access terminals can be increased horizontally by adding them to the service group through the configuration center. When there is a bottleneck in merging a single domain name on the storage side, the domain name can be sharded and the storage side can be increased horizontally. There is no bottleneck in the scheduling system, as long as high availability is achieved.
[0088] 7. System disaster recovery: When a node on the access end fails, the edge can remove it by configuring the central service discovery mechanism or timeout mechanism. When the scheduling system is down, ensure that an instance can successfully select the master through zk, and then check whether all the merges in the previous time period are successful. If not, re-execute the normal logic, require each node to send key information, and follow the normal command issuance logic. When the storage-server is down, restart it and wait for the master's command to execute. After receiving the command, execute the normal check logic to avoid repeated command execution.
[0089] Compared with some existing large-scale data set management models, the content distribution network log management system provided by this embodiment can avoid frequent data aggregation (shuffle), reduce the data transmission burden of the network and disk, and improve system performance.
[0090] The content distribution network log management system provided by this implementation can complete the global deduplication function of data at a very low cost; can effectively cope with sudden traffic situations of the system through horizontal expansion; can well cope with the complex business needs of users and can implement accurate billing logic.
[0091] The technical solution provided by one or more embodiments of the present disclosure can generate data keys through access-end components, mark initial log data, and form index log data, which is conducive to improving the efficiency of log data query and scheduling; through storage-end components, index log data can be transmitted and merged, and log data required by users can be sorted out, so as to facilitate the rapid and efficient generation of log files expected by users; through the task scheduling module, data merging requests can be disassembled from user access requests, and data merging requests can be distributed to storage-end components for processing, thereby flexibly meeting the different data access requirements of different users.
[0092] In addition, both the access-end components and the storage-end components can be expanded, ensuring the scalability of the management system. This is conducive to dealing with log data of different data volumes and can also effectively cope with traffic surges.
[0093] The present disclosure also provides a computer-readable storage medium, which is used to store a computer program. When the computer program is executed by a processor, the function of the access end component, storage end component or task scheduling module in the above-mentioned content distribution network log management system is realized.
[0094] Those skilled in the art can understand that the implementation of all or part of the functions of the access terminal component, storage terminal component or task scheduling module in the above-mentioned embodiment can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include all or part of the processes of the access terminal component, storage terminal component or task scheduling module. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.
[0095] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the embodiments of the device, equipment, and storage medium, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.
[0096] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
[0097] Although the embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.
Claims
1. A content distribution network log management system, characterized in that: The system comprises: an access-end module including at least one access-end component, a storage-end module including at least one storage-end component, and a task scheduling module; wherein, The access end component is used to receive and parse the initial log data; generate a data key corresponding to the initial log data according to the parsed information of the initial log data; mark the initial log data by the data key to form index log data; and send the index log data to the storage end component; The storage end component is used to receive and store the index log data; in response to the data merging request sent by the task scheduling module, transmit and merge the index log data between the storage end components; The task scheduling module is used to receive user query requests; generate at least one data merge request based on the user query request; and send the data merge request to the storage-end component so that the storage-end component transmits and merges the index log data based on the data merge request.
2. The system according to claim 1, characterized in that The parsing of the initial log data includes: Obtaining configuration information of a log database of the initial log data; Parsing the initial log data according to the configuration information to determine the domain name information of the initial log data; According to the configuration information, determining whether to process the initial log data in pieces; If the initial log data is processed by sharding, a shard value of the initial log data is generated.
3. The system according to claim 1, characterized in that The sending the index log data to the storage end component includes: Get the storage service list; Determine, according to the storage service list, a candidate storage component in the storage-end component; The index log data is sent to the candidate storage component.
4. The system according to claim 1, characterized in that The task scheduling module includes multiple task scheduling nodes, and the task scheduling nodes are divided into a master scheduling node and at least one slave scheduling node through a master selection mechanism; wherein the master scheduling node is used to receive the user's access request, generate the data merge request, and send the data merge request, and the slave scheduling node is used as a redundant guarantee for the master scheduling node.
5. The system according to claim 1, characterized in that Generating at least one data merging request according to the user's query request includes: At a preset time granularity, determine whether there is a data query request included in the user's query request in the current time segment; If the data query request exists in the current time segment, at least one data merge request is generated in response to the data query request.
6. The system according to claim 1 or 5, characterized in that: The generating at least one data merging request comprises: Traversing the storage-end component, obtaining the data key of the index log data and the component identifier of the storage-end component, and recording a matching relationship between the data key and the component identifier; Put the data key into a first memory queue to form a full set of keys; Putting the component identification into a second memory queue; For the first identifier in the second memory queue, a first subset key is extracted from the full set key, and a first merge request is generated. The first merge request is used to indicate the first storage component corresponding to the first identifier and merge the index log data corresponding to the first subset key.
7. The system according to claim 6, characterized in that The generating at least one data merging request comprises: If the first storage component corresponding to the first identifier has processed the data merging task indicated by the first merging request, the first identifier is put back into the second memory queue; Determine whether there are any remaining keys in the full set of keys in the first memory queue; If the remaining keys exist in the full set of keys, a second subset key is extracted from the remaining keys for the second identifier in the second memory queue, and a second merge request is generated, wherein the second merge request is used to indicate the second storage component corresponding to the second identifier, and merge the index log data corresponding to the second subset key.
8. The system according to claim 1, characterized in that The step of transmitting and merging the index log data between the storage end components in response to the data merging request sent by the task scheduling module includes: In response to the data merge request, the third storage component in the storage-end component determines a target key indicated by the data merge request and a target storage component matching the target key; The third storage component communicates with the target storage component to obtain the target log data corresponding to the target key in the target storage component; The third storage component merges the target log data.
9. The system according to claim 8, characterized in that The obtaining target log data corresponding to the target key in the target storage component includes: Identifying a target domain name included in the target key; For each of the target domain names, establish a Bloom filter; Obtain the target log data, and calculate the target hash value of the target log data; According to the domain name information of the target log data, the target hash value is placed into the corresponding Bloom filter to determine whether the target log data already exists; If the target log data already exists, reversal data of the target log data is generated, and the reversal data is used for billing.
10. The system according to claim 1, characterized in that The step of transmitting and merging the index log data between the storage end components in response to the data merging request sent by the task scheduling module includes: Check whether the data merging request has been processed; If the data merging request has been processed, return; If the data merging request has not been processed, checking whether the data merging request is being processed; If the data merging request is being processed, return; Otherwise, the transmission and merging of the index log data begins.