A method and system for tracking task deployment from center to edge nodes

Abstract

The present invention provides a task deployment tracking method and system from a center to an edge node, belonging to the field of IT and software development technology. The method comprises: firstly, an upstream system issues a refresh URL task; a central service unit receives the refresh URL task, generates a task ID, and stores task information in a database; then, an edge node service unit batch-pushes the refresh URL tasks that need to be processed by the node from the center according to the machine IP, performs machine content cleaning through a task processing module, and records the task status; then, a callback processing module receives the task processing results reported by the edge node service unit every minute, searches for the task bitmaps of all resource pools to which the IP belongs for that minute according to the machine IP of the edge node, performs exclusive OR calculations one by one on the reported bitmap and all bitmaps of the IP resource pool, and the exclusive OR result bitmap is a list of failed tasks of the machine IP of the edge node in that minute; finally, the failed task information of the edge node service unit is recorded, and a failed re-flush operation is performed.

Description

Technical Field

[0001] The present invention belongs to the technical field of content distribution network development, and in particular relates to a method and system for tracking task deployment from a center to an edge node. Background Art

[0002] CDN (Content Delivery Network) acceleration is a content delivery acceleration service based on a globally distributed network of nodes. It distributes source site content to the nodes closest to the user, enabling them to access the content they need locally. This addresses access latency issues caused by cross-carrier, cross-regional, and server bandwidth and performance issues, improving user access response speed and success rates. It is suitable for scenarios such as site acceleration, file downloads, and video on demand.

[0003] The CDN refresh feature forces the cached content of CDN nodes to expire after submitting a URL refresh request. When you request resources from a CDN node, the CDN will directly retrieve the corresponding resources from the origin server and cache them for you. This feature requires that all edge nodes' cached content be accurately cleared; failure to do so may result in users accessing corrupted data.

[0004] As CDN business scenarios become increasingly diverse, the number of URLs that need to be refreshed every day reaches hundreds of millions. Tracking the refresh results of URL cache content in tens of thousands of CDN nodes requires the central system to process trillions of edge node refresh callback messages every day, which is a huge consumption of machine resources and network bandwidth. Summary of the Invention

[0005] The purpose of the present invention is to provide a method and system for tracking task deployment from a center to an edge node, aiming to solve the problems mentioned in the background technology.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A method for tracking task deployment from a center to an edge node, the method comprising:

[0008] Step 1: The upstream system issues a refresh URL task;

[0009] Step 2: The task access module in the central service unit receives the refresh URL task in step 1, generates a task ID, and stores the task information in the database;

[0010] Step 3: The task data pulling module in the edge node service unit pulls the refresh URL tasks that need to be processed by the node from the center in batches according to the machine IP of the edge node, cleans up the machine content of the edge node through the task processing module, and records the task status;

[0011] Step 4: The callback processing module in the central service unit receives the minute-by-minute task processing results reported by the edge node service unit, including the node IP and minute timestamp, and the compressed bitmap of the task processing results. It uses the Roaring Bitmap algorithm to decompress the compressed bitmap to obtain the complete bitmap.

[0012] Step 5: Search the task bitmaps of all resource pools to which the edge node belongs for that minute based on the machine IP of the edge node. XOR the reported bitmap with all bitmaps of the IP resource pool one by one. The XOR result bitmap is the list of failed tasks of the machine IP of the edge node for that minute.

[0013] Step 6: Record the failed task information of the edge node service unit and perform a failed re-flash operation.

[0014] As a preferred solution of the present invention, in step 2, the task ID generation mechanism adopts a self-increment method. The byte has a total of 16 bits, the first ten bits are the current timestamp rounded to the minute value, and the last 6 bits are the self-increment count value. All task IDs of this minute start to increment from 1, and the maximum is 999999.

[0015] As a preferred solution of the present invention, the central service unit records the task list of each resource pool every minute with a bitmap, parses the minute timestamp and offset from the task ID, and sets the specific offset position to 1 on the bitmap of the corresponding minute.

[0016] As a preferred solution of the present invention, the edge node service unit uses a bitmap to record the task processing results every minute, parses the minute timestamp and offset from the task ID, and records 1 at the offset position of the bitmap if the task processing is successful, and records 0 if the task processing fails. After all tasks are processed in that minute, the bitmap fully records the results of all successful tasks.

[0017] As a preferred solution of the present invention, after completing task processing every minute, the edge node service unit generates a complete task status bitmap, compresses it using the Roaring Bitmap bitmap compression algorithm, and reports the minute timestamp and bitmap to the central service.

[0018] To implement the above method, the present invention provides a task deployment tracking system from the center to the edge node, which is specifically:

[0019] The upstream system unit is responsible for issuing the refresh URL task;

[0020] The edge node service unit aggregates all task results, compresses them using a custom compression algorithm, and then reports them to the central service unit.

[0021] The central service unit processes the refresh callback message of the edge node service unit.

[0022] As a preferred solution of the present invention, the southbound protocol refers to a protocol for transmitting configuration information, and the southbound protocol is used to feed back configuration delivery result information.

[0023] As a preferred solution of the present invention, the central service unit includes:

[0024] Basic data maintenance module, which maintains customer information, domain name information, resource pool information, IP information, mapping relationship between resource pool and IP, and mapping relationship between domain name and resource pool;

[0025] Task access module: The task access module is responsible for receiving refresh tasks submitted externally, generating task IDs, and storing task ID information in the database;

[0026] A callback processing module receives the processing result reported every minute by the edge node service unit.

[0027] As a preferred solution of the present invention, the edge node service unit is divided into multiple resource pools according to business, and one domain name will be assigned to one resource pool to provide services.

[0028] As a preferred solution of the present invention, the central service unit performs XOR calculation on the target task ID list in the callback processing module and the task access module, compares the failed task ID list of the edge node, and performs tracking processing.

[0029] Compared with the existing technology, in a scenario where 1 billion refresh tasks need to be deployed to 10,000 nodes every day, the original solution requires 16 characters for a refresh task to be called back from the edge to the center. After adopting the technical solution of the present invention, each task occupies 1 bit, and the common 16 characters per minute, the bandwidth consumption is reduced by nearly 128 times. Specifically, the total data volume using the original solution is 1280Tb, and the total data volume using the technical solution of the present invention is 10Tb, and the central service task message storage space is also reduced by 128 times. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0031] Figure 1 This is a block diagram of a task deployment tracking system from a center to an edge node of the present invention;

[0032] Figure 2 Schematic diagram of the task ID generation mechanism in the present invention;

[0033] Figure 3A schematic diagram of parsing minute timestamps and offsets from task IDs in the present invention;

[0034] Figure 4 This is a schematic diagram of a bitmap resulting from performing XOR operations on the bitmap reported by the edge node and all the bitmaps in the IP resource pool in the present invention. DETAILED DESCRIPTION

[0035] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0036] Example 1

[0037] See also Figure 1-4 , the present invention provides the following technical solutions:

[0038] A method for tracking task deployment from a center to an edge node, comprising:

[0039] Step 1: The upstream system issues a refresh URL task;

[0040] Step 2: The task access module in the central service unit receives the refresh URL task in step 1, generates a task ID, and stores the task information in the database;

[0041] Step 3: The task data pulling module in the edge node service unit pulls the refresh URL tasks that need to be processed by the node from the center in batches according to the machine IP of the edge node, cleans up the machine content of the edge node through the task processing module, and records the task status;

[0042] Step 4: The callback processing module in the central service unit receives the minute-by-minute task processing results reported by the edge node service unit, including the node IP and minute timestamp, and the compressed bitmap of the task processing results. It uses the Roaring Bitmap algorithm to decompress the compressed bitmap to obtain the complete bitmap.

[0043] It should be noted that the RoaringBitmap algorithm is a library for compressing bitmap data structures, designed to process large datasets quickly and efficiently. The RoaringBitmap algorithm divides the bitmap into several blocks, each containing several integer values. For each block, the RoaringBitmap algorithm uses a different encoding method for compression. For small blocks, dense encoding is used; for large blocks, sparse encoding is used. In dense encoding, the RoaringBitmap algorithm uses a bitmap to indicate whether all values ​​in the block exist. In sparse encoding, the RoaringBitmap algorithm uses an array to store the values ​​present in the block. The RoaringBitmap algorithm also supports operations such as union, intersection, and difference between multiple blocks. In these operations, the RoaringBitmap algorithm will select the encoding method that best suits the current operation to improve efficiency. The RoaringBitmap algorithm is an efficient compressed bitmap data structure suitable for processing large datasets. It improves efficiency by selecting the encoding method that best suits the current operation, and supports operations such as union, intersection, and difference between multiple blocks, making data processing more convenient and efficient. In this solution, the callback processing module processes the compressed bitmap through the RoaringBitmap algorithm.

[0044] Step 5: Search the task bitmaps of all resource pools to which the edge node belongs for that minute based on the machine IP of the edge node. XOR the reported bitmap with all bitmaps of the IP resource pool one by one. The XOR result bitmap is the list of failed tasks of the machine IP of the edge node for that minute.

[0045] Step 6: Record the failed task information of the edge node service unit and perform a failed re-flash operation.

[0046] It should be noted that the refresh URL task in this method can also be URL blocking and unblocking, pre-fetching, live stream control blocking and interruption, regular command deployment and execution, etc.

[0047] In a specific embodiment of the present invention, a custom algorithm is used to generate task IDs. Tasks in the same minute all use the same prefix. Each resource pool in the central service unit uses a bitmap to store a one-minute task list. After upstream system tasks are sent in batches to the edge node service units, each edge node service unit summarizes all task results for the minute every minute, compresses them using a custom compression algorithm, and then reports them to the central service unit. The central service unit uses an XOR calculation with the target task ID list to compare the failed task ID list for the node and perform tracking processing, thereby tracking the refresh results of each refresh URL task in all edge node service units. This also reduces network transmission overhead and the resource overhead of the central comparison calculation.

[0048] For details, please refer to Figure 2 The task ID generation mechanism adopts the self-increment method. The task ID has 16 bits in total. The first ten bits are the current timestamp rounded to the minute value, and the last six bits are the self-increment count value. All task IDs in that minute start to increase from 1 and the maximum value is 999999.

[0049] For details, please refer to Figure 3 ,The task list of each resource pool every minute in the central service unit is recorded in a bitmap. The minute timestamp and offset are parsed from the target task ID. The specific offset position is set to 1 on the bitmap of the corresponding minute.

[0050] Specifically, the edge node service unit uses a bitmap to record task processing results every minute. The minute timestamp and offset are parsed from the target task ID. Successful task processing is recorded at the bitmap offset as 1, while failure is recorded as 0. After all tasks are processed for that minute, the bitmap fully records the results of all successful tasks. After completing all tasks for that minute, a complete task status bitmap is generated, compressed using the Roaring Bitmap bitmap compression algorithm, and then reported to the central service with the minute timestamp and bitmap.

[0051] For details, please refer to Figure 4 IP1 belongs to both resource pool 1 and resource pool 2. IP1 needs to process all tasks of resource pool 1 and resource pool 2 every minute. After the bitmap data reported by IP1 is XORed with the bitmaps of resource pool 1 and resource pool 2, the XOR values ​​of the green and purple parts are 0, and the XOR value of the remaining red parts with offsets of 5 and 8 is 1, indicating failure. The tasks with task IDs 1684414800000005 and 1684414800000008 are considered failed.

[0052] See Figure 1 Another embodiment of the present application provides a task deployment tracking system from a center to an edge node, the system comprising:

[0053] The upstream system unit is responsible for issuing the refresh URL task;

[0054] The edge node service unit aggregates all task results, compresses them using a custom compression algorithm, and then reports them to the central service unit.

[0055] The central service unit processes the refresh callback message of the edge node service unit.

[0056] Specifically, the edge node service unit includes a task data pull module, a task processing module, and a result callback module. The task data pull module batch-pull refresh URL tasks that the node needs to process from the center based on the edge node machine IP address. The task processing module cleans up the edge node machine's content and records the task status. After the minute task is processed, the result callback module generates a complete task status bitmap. After compressing it using the Roaring Bitmap bitmap compression algorithm, it reports the minute timestamp and bitmap to the callback processing module of the central service unit.

[0057] An embodiment of the present application further provides a readable storage medium, which may store a program suitable for execution by a processor, wherein the program is used to implement various processing flows of the aforementioned terminal in a hybrid cloud adaptive resource management solution.

[0058] An embodiment of the present invention further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any one of the above method embodiments.

[0059] It should be noted that for the aforementioned method embodiments, for simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should be aware that the present invention is not limited by the order of the actions described, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0060] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0061] In the several embodiments provided by the present invention, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the above-mentioned units is merely a logical functional division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interface, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0062] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0063] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0064] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a memory and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the above-mentioned methods of each embodiment of the present invention. The aforementioned memory includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0065] The embodiments of the present invention are described in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

Claims

1. A method for tracking task deployment from a center to an edge node, characterized in that: include: Step 1: The upstream system issues a refresh URL task; Step 2: The task access module in the central service unit receives the refresh URL task in step 1, generates a task ID, and stores the task information in the database; Step 3: The task data pulling module in the edge node service unit pulls the refresh URL tasks that need to be processed by the node from the center in batches according to the machine IP of the edge node, cleans up the machine content of the edge node through the task processing module, and records the task status; Step 4: The callback processing module in the central service unit receives the minute-by-minute task processing results reported by the edge node service unit, including the node IP and minute timestamp, and the compressed bitmap of the task processing results. It uses the Roaring Bitmap algorithm to decompress the compressed bitmap to obtain the complete bitmap. Step 5: Find the task bitmaps for that minute in all resource pools to which the edge node belongs based on the machine IP of the edge node. XOR the reported bitmap with all the bitmaps in the IP resource pool one by one. The XOR result bitmap is the list of failed tasks for the machine IP of the edge node in that minute. Step 6: Record the failed task information of the edge node service unit and perform a failed re-flash operation.

2. The method for tracking task deployment from a center to an edge node according to claim 1, characterized in that: In step 2, the task ID generation mechanism adopts a self-incrementing method. The byte has a total of 16 bits, the first ten bits are the current timestamp rounded to the minute value, and the last 6 bits are the self-incrementing count value. All task IDs of the minute start to increase from 1 to a maximum of 999999.

3. The method for tracking task deployment from a center to an edge node according to claim 2, wherein: The central service unit records the task list of each resource pool every minute with a bitmap, parses the minute timestamp and offset from the task ID, and sets the specific offset position to 1 on the bitmap corresponding to the minute.

4. The method for tracking task deployment from a center to an edge node according to claim 3, wherein: The edge node service unit uses a bitmap to record the task processing results every minute, parses the minute timestamp and offset from the task ID, records 1 at the offset position of the bitmap if the task processing is successful, and records 0 if the task processing fails. After all tasks are processed in that minute, the bitmap fully records the results of all successful tasks.

5. The method for tracking task deployment from a center to an edge node according to claim 4, characterized in that: After completing the task processing every minute, the edge node service unit generates a complete task status bitmap, compresses it using the RoaringBitmap bitmap compression algorithm, and reports the minute timestamp and bitmap to the central service.

6. A task deployment tracking system from center to edge node, characterized in that: The system comprises: The upstream system unit is responsible for issuing the refresh URL task; The edge node service unit aggregates all task results, compresses them using a custom compression algorithm, and then reports them to the central service unit. The central service unit processes the refresh callback message of the edge node service unit.

7. The task deployment tracking system from center to edge node according to claim 6, characterized in that: The central service unit comprises: Basic data maintenance module, which maintains customer information, domain name information, resource pool information, IP information, mapping relationship between resource pool and IP, and mapping relationship between domain name and resource pool; Task access module: The task access module is responsible for receiving refresh tasks submitted externally, generating task IDs, and storing task ID information in the database; A callback processing module receives the processing result reported every minute by the edge node service unit.

8. The task deployment tracking system from center to edge node according to claim 6, characterized in that: The edge node service unit is divided into multiple resource pools according to business, and one domain name will be assigned to one resource pool to provide services.

9. The task deployment tracking system from center to edge node according to claim 8, characterized in that: The URL refresh request of the domain name requires all nodes in the resource pool providing the domain name service to execute cache content refresh.

10. The task deployment tracking system from center to edge node according to claim 6, characterized in that: The central service unit performs XOR calculation between the callback processing module and the target task ID list in the task access module, compares the failed task ID list of the edge node, and performs tracking processing.

Images