CDN content refresh system and method
Through the design of the CDN content refresh system, the problem of inconsistency in the refresh task execution process in the CDN system is solved, timely update of content and business stability are achieved, and the quality of CDN services and user experience are ensured.
Patent Information
- Application Number
- CN202010827274.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-08-17
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2040-08-17
AI Technical Summary
In the CDN system, during the issuance and execution of refresh tasks, due to the instability of heterogeneous systems and devices, the content of the CDN node is often inconsistent with the content of the source site, which affects the normal operation of the service and even causes user losses.
Provide a CDN content refresh system, including refresh task interface, monitor, scanner, content grabber, comparator and corrector, through periodic monitoring and analysis of refresh task information, scan CDN service nodes, grab and compare sources The contents of the station and CDN nodes are discovered in a timely manner and processed to ensure the consistency of the content.
It realizes timely discovery and elimination of inconsistencies between CDN nodes and source station content, ensures normal business operation, ensures CDN service quality and user experience, and is lightweight in deployment, low resource consumption, and supports flexible content comparison strategies and task monitoring.
Smart Images

Figure CN111949902B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of CDN content refresh, and in particular to a CDN content refresh system and method. Background Art
[0002] With the development of CDN, more and more CDN system platforms have been connected and collaborated with each other. CDN services are accessed through a unified business system, and the services are actually scheduled and allocated to run on multiple CDN system platforms. The internal implementations and mechanisms of these CDN systems are different. In other words, although the business capabilities output by each CDN system are basically the same, the underlying systems that undertake the business are heterogeneous.
[0003] The CDN system itself is a distributed system. The system structure is characterized by numerous subsystems, modules, nodes and devices. The output of a certain function requires high availability of each subsystem, module, node and device to ensure that the function is executed as expected.
[0004] For businesses running on multiple heterogeneous CDN systems, the probability of functions not being executed correctly as expected is higher than that of a single CDN vendor, which will affect the quality and experience of CDN services.
[0005] Although the design and implementation of the CDN system is a highly available distributed system, the CDN system has many systems, modules, and process links. The upgrade and changes of system modules, business configuration, equipment changes, etc. increase the instability of the system. Business loopholes exist and the probability of their occurrence is relatively high. Any problem in any system, module or link will cause a certain function to not be executed as expected. In the CDN business, refresh is a function that is used frequently in daily life. CDN users need to use the refresh function provided by the CDN manufacturer to ensure that the content in the CDN is consistent with the source site to ensure the normal operation of the business. For user businesses that are highly sensitive to content inconsistency, if the content in the CDN is inconsistent with the content of the source site and exceeds a certain period of time, it will cause service failures for users, affecting the normal operation of their business and even causing losses.
[0006] In the implementation, after the CDN user submits the refresh task, the system often shows that the delivery and execution are successful during delivery and execution. However, the user's business loss often occurs because the content in a certain node in the CDN system is inconsistent with the content of the source station. For user businesses that need to be refreshed frequently, the probability of such problems will be higher, and even affect the normal operation of the user's business. The main reason for such problems is that the refresh delivery execution target is the service device, and the CDN service devices are numerous, frequently changing and distributed in the network. Any device failure, network problem, system bug and configuration change may cause the service function execution failure of the device. Summary of the invention
[0007] The present invention provides a CDN content refresh system and method, which can overcome certain defects of the prior art.
[0008] The present invention provides a CDN content refresh system, which includes:
[0009] Refresh task interface, used to refresh tasks and obtain refresh task information issued by customers;
[0010] Refresh task monitor, used to periodically monitor refresh task information and analyze and resolve refresh task information;
[0011] CDN service node scanner, used to scan CDN service node information;
[0012] Node content crawler, used to access CDN service nodes or origin stations and save the crawled content;
[0013] Refresh the content comparator to compare the source server content and the service node content using the comparison strategy;
[0014] The refresh task modifier is used to perform preset actions based on the comparison results between the source station content and the service node content.
[0015] The present invention also provides a CDN content refreshing method, which comprises the following steps:
[0016] 1: Refresh task, obtain the refresh task information issued by the customer;
[0017] 2. Analyze and resolve the refresh task information, extract the refreshed domain name, URL and target CDN vendor information;
[0018] 3. Obtain the customer's origin site information and cache strategy configuration information by querying the configuration system;
[0019] 4. Scan and refresh the service node information related to the task according to the target CDN manufacturer information;
[0020] 5. Simulate the terminal user to crawl the content corresponding to the URL in the origin site, and at the same time crawl the content of the CDN vendor service node corresponding to the URL in sequence;
[0021] 6. Select the comparison strategy in the cache strategy configuration information, compare the content captured in the CDN vendor service node with the content of the source site, and determine whether the content is consistent. If all are consistent, end directly; if not, proceed to the next step;
[0022] 7. Record inconsistent service nodes and their captured content, and perform refresh operations or issue alarm operations according to preset processing actions.
[0023] Among them, in step 1, refresh the task while monitoring the refresh task.
[0024] The monitoring is to monitor the execution results of the refresh tasks with specific characteristics within a period of time. The refresh tasks with specific characteristics can be refresh tasks submitted by users, or refresh tasks related to domain names, or refresh tasks related to URL content.
[0025] Among them, the monitoring will maintain a refresh task queue that needs to be monitored. Refresh tasks that do not need to be monitored will not be included in the list. If a new task is submitted and the characteristics of the task are the same as those of the previous task that needs to be monitored, the old monitoring task will be overwritten; the execution result of the monitoring task will not be started until the refresh task is issued and marked as completed.
[0026] Among them, in step 4, the scan can be performed by using the extended DNS protocol through the authoritative DNS of dig CDN to obtain the service IP address resolved by the CDN manufacturer in each area for scanning, or by obtaining the service node corresponding to the URL through the full IP interface provided by the CDN manufacturer for scanning.
[0027] Among them, in step five, before crawling, the simulated terminal user needs to simulate the generation of a legal URL through custom configuration to pass the specific anti-hotlink verification, and can access multiple CDN nodes concurrently.
[0028] Among them, in step six, the comparison content is one or more of Last-Modified, ETag, and file MD5.
[0029] Among them, in step seven, the alarm operation can be sent by sending email, QQ, or WeChat so that operation and maintenance personnel can access and handle it.
[0030] In view of the problems and difficulties existing in CDN refresh, the present invention provides a method and system, which can timely discover the inconsistency between the content in the CDN service node and the source station, and update the content in the user's latest task to the node device in the CDN, and timely eliminate the inconsistency between the CDN node content and the source station content, thereby ensuring the normal operation of the business and guaranteeing the quality and experience of CDN services.
[0031] The lightweight deployment of the present invention can be deployed only for specific businesses, reducing unnecessary resource consumption. The plug-in content comparison strategy can be used to follow up on business selection of different strategies, and refresh tasks can be tracked and monitored periodically to ensure that the tasks are executed and take effect on all devices. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 This is a structural block diagram of a CDN content refresh system in Example 1;
[0033] Figure 2 This is a flowchart of a CDN content refresh method in Example 1. DETAILED DESCRIPTION
[0034] In order to further understand the content of the present invention, the present invention is described in detail in conjunction with the accompanying drawings and embodiments. It should be understood that the embodiments are only for explaining the present invention and are not intended to limit it.
[0035] Example 1
[0036] like Figure 1 As shown, this embodiment provides a CDN content refresh system, which includes:
[0037] Refresh task interface, used to refresh tasks and obtain refresh task information issued by customers;
[0038] Refresh task monitor, used to periodically monitor refresh task information and analyze and resolve refresh task information;
[0039] CDN service node scanner, used to scan CDN service node information;
[0040] Node content crawler, used to access CDN service nodes or origin stations and save the crawled content;
[0041] Refresh the content comparator to compare the source server content and the service node content using the comparison strategy;
[0042] The refresh task modifier is used to perform preset actions based on the comparison results between the source station content and the service node content.
[0043] The monitor periodically traverses the refresh tasks, checks the refresh URL tasks submitted in the recent period, and monitors the execution results of refresh tasks with specific characteristics within a period of time. For example, the execution results of refresh tasks will be monitored for 24 hours or even several days to ensure that within this period of time, the content on all CDN nodes used for service is consistent with the content on the refresh tasks.
[0044] like Figure 2 As shown, this embodiment provides a CDN content refresh method, which includes the following steps:
[0045] 1: Refresh task, obtain the refresh task information issued by the customer;
[0046] 2. Analyze and resolve the refresh task information, extract the refreshed domain name, URL and target CDN vendor information;
[0047] 3. Obtain the customer's origin site information and cache strategy configuration information by querying the configuration system;
[0048] 4. Scan and refresh the service node information related to the task according to the target CDN manufacturer information;
[0049] Five: Simulate the terminal user to crawl the content corresponding to the URL in the origin site, and at the same time crawl the content of the CDN vendor service node corresponding to the URL in sequence; simulate the user's normal access behavior, access the CDN service node or the origin site, and save the crawled content, usually including the http response header file itself;
[0050] 6. Select the comparison strategy in the cache strategy configuration information, compare the content captured in the CDN vendor service node with the content of the source site, and determine whether the content is consistent. If all are consistent, end directly; if not, proceed to the next step;
[0051] 7. Record inconsistent service nodes and their captured content, and perform refresh operations or issue alarm operations according to preset processing actions.
[0052] Among them, in step 1, refresh the task while monitoring the refresh task.
[0053] The monitoring is to monitor the execution results of the refresh tasks with specific characteristics within a period of time. The refresh tasks with specific characteristics can be refresh tasks submitted by users, or refresh tasks related to domain names, or refresh tasks related to URL content.
[0054] Among them, the monitoring will maintain a refresh task queue that needs to be monitored, and refresh tasks that do not need to be monitored will not be included in the list, which can avoid unnecessary processing and reduce the amount of data processing. If a new task is submitted, and the characteristics of the task are the same as those of the task that needed to be monitored before, then the old monitoring task will be overwritten. For example, three days ago, a user submitted a refresh task for a URL that was monitored. Today, the customer resubmitted a refresh task for the same URL, then the new task will overwrite the old task and reset the monitoring time of this URL task; the execution result of the monitoring task will not be started until the refresh task is sent and marked as completed, thus ensuring that the refresh task has been sent to the CDN system and executed.
[0055] Among them, in step 4, the scan can be performed by using the extended DNS protocol through the authoritative DNS of dig CDN to obtain the service IP address resolved by the CDN manufacturer in each region for scanning, or by obtaining the service node corresponding to the URL through the full IP interface provided by the CDN manufacturer for scanning. The service IP address of the CDN will be dynamically adjusted due to factors such as equipment failure, performance and operation, and may change over a period of time. Usually, the URL corresponding to the full network resolution IP address is re-obtained at the beginning of each URL check.
[0056] Among them, in step five, before crawling, the simulated terminal user needs to simulate the generation of a legitimate URL through custom configuration to pass the specific anti-hotlink verification, and can access multiple CDN nodes concurrently at the same time, thus improving the detection speed.
[0057] Among them, in step 6, the comparison content is one or more of Last-Modified, ETag, and file MD5. The comparator needs to crawl the content of the source station and also needs to crawl the content on each service node. The comparator uses the comparison strategy to compare the source station content and the service node content, and outputs the comparison result according to the strategy.
[0058] Among them, in step seven, the alarm operation can be sent by sending email, QQ, or WeChat so that operation and maintenance personnel can access and handle it.
[0059] The present invention and its embodiments are described schematically above, and the description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. Therefore, if a person skilled in the art is inspired by it and designs a structural method and an embodiment similar to the technical solution without creativity without departing from the purpose of the invention, they shall all fall within the protection scope of the present invention.
Claims
1. A CDN content refresh system, characterized by: Includes the following connected in sequence: Refresh task interface, used to refresh tasks and obtain refresh task information issued by customers; Refresh task monitor, used to periodically monitor refresh task information and analyze and resolve refresh task information; CDN service node scanner, used to scan CDN service node information; Node content crawler, used to access CDN service nodes or origin stations and save the crawled content; Refresh the content comparator, which is used to compare the source site content and the CDN service node content using the comparison strategy: if all are consistent, it ends directly; If the content is inconsistent, proceed to the next step; The refresh task corrector is used to perform preset actions based on the comparison results between the source site content and the CDN service node content: record inconsistent service nodes and their crawled content, and perform a refresh operation or issue an alarm operation based on the preset processing action.
2. A CDN content refresh method, characterized by: The following steps are involved: 1: Refresh task, obtain the refresh task information issued by the customer; 2. Analyze and resolve the refresh task information, extract the refreshed domain name, URL and target CDN vendor information; 3. Obtain the customer's origin site information and cache strategy configuration information by querying the configuration system; 4. Scan and refresh the service node information related to the task according to the target CDN manufacturer information; 5. Simulate the terminal user to crawl the content corresponding to the URL in the origin site, and at the same time crawl the content of the CDN vendor service node corresponding to the URL in sequence; 6. Select the comparison strategy in the cache strategy configuration information, compare the content captured by the CDN vendor service node with the content of the source site, and determine whether the content is consistent. If all are consistent, end directly; If the content is inconsistent, proceed to the next step; 7. Record inconsistent service nodes and their captured content, and perform refresh operations or issue alarm operations according to preset processing actions.
3. A CDN content refresh method according to claim 2, characterized in that: In step 1, refresh the task while monitoring the refresh task.
4. A CDN content refresh method according to claim 3, characterized in that: Monitoring is to monitor the execution results of refresh tasks with specific characteristics within a period of time. Refresh tasks with specific characteristics are refresh tasks submitted by users, refresh tasks related to domain names, or refresh tasks related to URL content.
5. A CDN content refresh method according to claim 4, characterized in that: The monitoring system will maintain a refresh task queue that needs to be monitored. Refresh tasks that do not need to be monitored will not enter the queue. If a new task is submitted and its characteristics are the same as those of the previous task that needs to be monitored, the old monitoring task will be overwritten. The execution result of the monitoring task will not be started until the refresh task is issued and marked as completed.
6. A CDN content refresh method according to claim 2, characterized in that: In step 4, the scan can be performed by using the extended DNS protocol through digCDN's authoritative DNS to obtain the service IP addresses resolved by the CDN vendor in each region for scanning, or by using the full IP interface provided by the CDN vendor to obtain the service node corresponding to the URL for scanning.
7. A CDN content refresh method according to claim 2, characterized in that: In step 5, before crawling, the simulated terminal user needs to simulate the generation of a legitimate URL through custom configuration to pass the anti-hotlink verification, and can access multiple CDN nodes concurrently.
8. A CDN content refresh method according to claim 2, characterized in that: In step 6, the comparison content is one or more of Last-Modified, ETag, and file MD5.
9. A CDN content refresh method according to claim 2, characterized in that: In step seven, the alarm operation can be sent by sending email, QQ, or WeChat so that operation and maintenance personnel can access and handle it.
