Task deployment tracking method and system from center to edge node

Abstract

The present application relates to the technical field of IT and software development, and relates to a task deployment tracking method and system from a center to an edge node. The method comprises: an upstream system first issues URL refresh tasks; a center service unit receives the URL refresh tasks, generates task IDs, and stores task information to a database; then an edge node service unit pulls in batches, from the center and on the basis of a machine IP, URL refresh tasks needing to be processed by the node, executes cleaning of machine content by means of a task processing module, and records a task state; then a callback processing module receives task processing results reported by the edge node service unit every minute, and then on the basis of the machine IP of the edge node, searches for a task bitmap of all resource pools to which the IP belongs for that minute, a reported bitmap undergoes XOR computation with all bitmaps of the IP resource pools one by one, and an XOR result bitmap is a list of failed tasks for the machine IP of the edge node in that minute; and finally, failed task information of the edge node service unit is recorded, and a failure refresh retry operation is performed.

Description

A method and system for tracking task deployment from center to edge nodes

[0001] Related applications

[0002] This application claims priority to Chinese patent application number 2023117126487, filed on December 13, 2023, entitled “A method and system for tracking task deployment from center to edge nodes,” the entire text of which is incorporated herein by reference. Technical Field

[0003] The present application belongs to the technical field of content distribution network development, and specifically relates to a method and system for tracking task deployment from a center to an edge node. Background Art

[0004] CDN (Content Delivery Network) acceleration is a content delivery acceleration service based on a globally distributed network of nodes. It distributes source site content to the nodes closest to the user, enabling them to access the content they need locally. This addresses access latency issues caused by cross-carrier, cross-regional, and server bandwidth and performance issues, improving user access response speed and success rates. It is suitable for scenarios such as site acceleration, file downloads, and video on demand.

[0005] The CDN refresh feature forces the cached content of CDN nodes to expire after submitting a URL refresh request. When you request resources from a CDN node, the CDN will directly retrieve the corresponding resources from the origin server and cache them for you. This feature requires that all edge nodes' cached content be accurately cleared; failure to do so may result in users accessing corrupted data.

[0006] As CDN business scenarios become increasingly diverse, the number of URLs that need to be refreshed every day reaches hundreds of millions. Tracking the refresh results of URL cache content in tens of thousands of CDN nodes requires the central system to process trillions of edge node refresh callback messages every day, which is a huge consumption of machine resources and network bandwidth. Summary of the Invention

[0007] The purpose of this application is to provide a method and system for tracking task deployment from a center to an edge node, aiming to solve the problems mentioned in the background technology.

[0008] To achieve the above objectives, this application provides the following technical solutions:

[0009] A method for tracking task deployment from a center to an edge node, the method comprising:

[0010] Step 1: The upstream system issues a refresh URL task;

[0011] Step 2: The task access module in the central service unit receives the refresh URL task in step 1, generates a task ID, and stores the task information in the database;

[0012] Step 3: The task data pulling module in the edge node service unit pulls the refresh URL tasks that need to be processed by the node from the center in batches according to the machine IP of the edge node, cleans up the machine content of the edge node through the task processing module, and records the task status;

[0013] Step 4: The callback processing module in the central service unit receives the minute-by-minute task processing results reported by the edge node service unit, including the node IP and minute timestamp, and the compressed bitmap of the task processing results. It uses the Roaring Bitmap algorithm to decompress the compressed bitmap to obtain the complete bitmap.

[0014] Step 5: Find the task bitmaps for that minute in all resource pools to which the edge node belongs based on the machine IP of the edge node. XOR the reported bitmap with all the bitmaps in the IP resource pool one by one. The XOR result bitmap is the list of failed tasks for the machine IP of the edge node in that minute.

[0015] Step 6: Record the failed task information of the edge node service unit and perform a failed re-flash operation.

[0016] As a preferred solution of the present application, in step 2, the task ID generation mechanism adopts a self-incrementing method. The byte has a total of 16 bits, the first ten digits are the current timestamp rounded to the minute value, and the last 6 digits are the self-incrementing count value. All task IDs of this minute start to increment from 1, and the maximum is 999999.

[0017] As a preferred solution of the present application, the central service unit records the task list of each resource pool every minute with a bitmap, parses the minute timestamp and offset from the task ID, and sets the specific offset position to 1 on the bitmap of the corresponding minute.

[0018] As a preferred solution of the present application, the edge node service unit uses a bitmap to record the task processing results every minute, parses the minute timestamp and offset from the task ID, and records 1 at the offset position of the bitmap if the task processing is successful, and records 0 if the task processing fails. After all tasks are processed in that minute, the bitmap fully records the results of all successful tasks.

[0019] As a preferred solution of the present application, after completing the task processing every minute, the edge node service unit generates a complete task status bitmap, compresses it using the Roaring Bitmap bitmap compression algorithm, and reports the minute timestamp and bitmap to the central service.

[0020] To implement the above method, this application provides a task deployment tracking system from the center to the edge node, the specific system is:

[0021] The upstream system unit is responsible for issuing the refresh URL task;

[0022] The edge node service unit aggregates all task results, compresses them using a custom compression algorithm, and then reports them to the central service unit.

[0023] The central service unit processes the refresh callback message of the edge node service unit.

[0024] As a preferred solution of the present application, the southbound protocol refers to a protocol for transmitting configuration information, and the southbound protocol is used to feed back configuration delivery result information.

[0025] As a preferred solution of this application, the central service unit includes:

[0026] A basic data maintenance module, which maintains customer information, domain name information, resource pool information, IP information, mapping relationships between resource pools and IP addresses, and mapping relationships between domain names and resource pools;

[0027] The task access module is responsible for receiving refresh tasks submitted externally, generating task IDs, and storing task ID information in the database;

[0028] A callback processing module receives the processing result reported every minute by the edge node service unit.

[0029] As a preferred solution of the present application, the edge node service unit is divided into multiple resource pools according to business, and one domain name will be assigned to one resource pool to provide services.

[0030] As a preferred solution of the present application, the central service unit performs XOR calculation on the target task ID list in the callback processing module and the task access module, compares the failed task ID list of the edge node, and performs tracking processing.

[0031] Compared with traditional methods, in a scenario where 1 billion refresh tasks need to be deployed to 10,000 nodes every day, the original solution requires 16 characters for a refresh task to be called back from the edge to the center. After adopting the technical solution of the present application, each task occupies 1 bit, and the common 16 characters per minute, the bandwidth consumption is reduced by nearly 128 times. Specifically, the total data volume using the original solution is 1280Tb, and the total data volume using the technical solution of the present application is 10Tb, and the central service task message storage space is also reduced by 128 times. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the conventional technology, the following briefly introduces the drawings required for use in the embodiments or the conventional technology descriptions. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the disclosed drawings without any creative work.

[0033] FIG1 is a block diagram of a task deployment tracking system from a center to an edge node in the present application;

[0034] Figure 2 is a schematic diagram of the task ID generation mechanism in this application;

[0035] FIG3 is a schematic diagram of parsing minute timestamps and offsets from task IDs in this application;

[0036] FIG4 is a schematic diagram of a bitmap obtained by performing XOR operations on the bitmap reported by the edge node and all the bitmaps in the IP resource pool in this application. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0038] Example 1

[0039] [Corrected 06.01.2025 according to Rule 91] Referring to Figures 1-4, this application provides the following technical solutions:

[0040] A method for tracking task deployment from a center to an edge node, comprising:

[0041] Step 1: The upstream system issues a refresh URL task;

[0042] Step 2: The task access module in the central service unit receives the refresh URL task in step 1, generates a task ID, and stores the task information in the database;

[0043] Step 3: The task data pulling module in the edge node service unit pulls the refresh URL tasks that need to be processed by the node from the center in batches according to the machine IP of the edge node, cleans up the machine content of the edge node through the task processing module, and records the task status;

[0044] Step 4: The callback processing module in the central service unit receives the minute-by-minute task processing results reported by the edge node service unit, including the node IP and minute timestamp, and the compressed bitmap of the task processing results. It uses the Roaring Bitmap algorithm to decompress the compressed bitmap to obtain the complete bitmap.

[0045] It should be noted that the RoaringBitmap algorithm is a library for compressing bitmap data structures, designed to process large datasets quickly and efficiently. The RoaringBitmap algorithm divides the bitmap into several blocks, each containing several integer values. For each block, the RoaringBitmap algorithm uses a different encoding method for compression. For small blocks, dense encoding is used; for large blocks, sparse encoding is used. In dense encoding, the RoaringBitmap algorithm uses a bitmap to indicate whether all values ​​in the block exist. In sparse encoding, the RoaringBitmap algorithm uses an array to store the values ​​present in the block. The RoaringBitmap algorithm also supports operations such as union, intersection, and difference between multiple blocks. In these operations, the RoaringBitmap algorithm selects the encoding method that best suits the current operation to improve efficiency. The RoaringBitmap algorithm is an efficient compressed bitmap data structure suitable for processing large datasets. It improves efficiency by selecting the encoding method that best suits the current operation, and supports operations such as union, intersection, and difference between multiple blocks, making data processing more convenient and efficient. In this solution, the callback processing module processes the compressed bitmap through the RoaringBitmap algorithm.

[0046] Step 5: Search the task bitmaps of all resource pools to which the edge node belongs for that minute based on the machine IP of the edge node. XOR the reported bitmap with all bitmaps of the IP resource pool one by one. The XOR result bitmap is the list of failed tasks of the machine IP of the edge node for that minute.

[0047] Step 6: Record the failed task information of the edge node service unit and perform a failed re-flash operation.

[0048] It should be noted that the refresh URL task in this method can also be URL blocking and unblocking, pre-fetching, live stream control blocking and interruption, regular command deployment and execution, etc.

[0049] In a specific embodiment of the present application, a task ID is generated by a custom algorithm. Tasks in the same minute all use the same prefix. Each resource pool of the central service unit uses a bitmap to store a one-minute task list. After the upstream system tasks are sent in batches to the edge node service units, each edge node service unit summarizes all task results for one minute every minute, compresses them using a custom compression algorithm, and then reports them to the central service unit. The central service unit uses an XOR calculation with the target task ID list to compare the failed task ID list of the node and perform tracking processing to track the refresh results of each refresh URL task in all edge node service units. At the same time, it reduces the network transmission overhead and the resource overhead of the central comparison calculation.

[0050] Please refer to Figure 2 for details. The task ID generation mechanism adopts a self-increasing method. The task ID byte has a total of 16 bits. The first ten bits are the current timestamp rounded to the minute value, and the last six bits are the self-incrementing count value. All task IDs in that minute start from 1 and increase automatically, with a maximum of 999999.

[0051] For details, please refer to Figure 3. The task list of each resource pool per minute in the central service unit is recorded in a bitmap. The minute timestamp and offset are parsed from the target task ID. On the bitmap corresponding to the minute, the specific offset position is set to 1.

[0052] Specifically, the edge node service unit uses a bitmap to record task processing results every minute. The minute timestamp and offset are parsed from the target task ID. Successful task processing is recorded at the bitmap offset as 1, while failure is recorded as 0. After all tasks are processed for that minute, the bitmap fully records the results of all successful tasks. After completing all tasks for that minute, a complete task status bitmap is generated, compressed using the Roaring Bitmap bitmap compression algorithm, and then reported to the central service with the minute timestamp and bitmap.

[0053] Please refer to Figure 4 for details. IP1 belongs to both resource pool 1 and resource pool 2. IP1 needs to process all tasks of resource pool 1 and resource pool 2 every minute. After the bitmap data reported by IP1 is XORed with the bitmaps of resource pool 1 and resource pool 2, the XOR values ​​of the green and purple parts are 0, and the XOR value of the red parts with the remaining offsets of 5 and 8 is 1, indicating failure. This converts to tasks with task IDs 1684414800000005 and 1684414800000008, which are considered failed.

[0054] Referring to FIG1 , another embodiment of the present application provides a task deployment tracking system from a center to an edge node, the system comprising:

[0055] The upstream system unit is responsible for issuing the refresh URL task;

[0056] The edge node service unit aggregates all task results, compresses them using a custom compression algorithm, and then reports them to the central service unit.

[0057] The central service unit processes the refresh callback message of the edge node service unit.

[0058] Specifically, the edge node service unit includes a task data pull module, a task processing module, and a result callback module. The task data pull module batch-pull refresh URL tasks that the node needs to process from the center based on the edge node machine IP address. The task processing module cleans up the edge node machine's content and records the task status. After the minute task is processed, the result callback module generates a complete task status bitmap. After compressing it using the Roaring Bitmap bitmap compression algorithm, it reports the minute timestamp and bitmap to the callback processing module of the central service unit.

[0059] An embodiment of the present application further provides a readable storage medium, which may store a program suitable for execution by a processor, wherein the program is used to implement various processing flows of the aforementioned terminal in a hybrid cloud adaptive resource management solution.

[0060] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any one of the above method embodiments.

[0061] It should be noted that for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0062] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0063] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0064] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0065] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0066] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the traditional method or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory, including a number of instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the above-mentioned methods of each embodiment of the present application. The aforementioned memory includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0067] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0068] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A task deployment tracking method from a center to an edge node, characterized in that: include: Step 1: The upstream system issues a refresh URL task; Step 2: The task access module in the central service unit receives the refresh URL task in step 1, generates a task ID, and stores the task information in the database; Step 3: The task data pulling module in the edge node service unit pulls the refresh URL tasks that need to be processed by the node from the center in batches according to the machine IP of the edge node, cleans up the machine content of the edge node through the task processing module, and records the task status; Step 4: The callback processing module in the central service unit receives the task processing results reported by the edge node service unit every minute, including the node IP and minute timestamp, and the compressed bitmap of the task processing results, and uses the Roaring Bitmap algorithm to decompress it to obtain the complete bitmap; Step 5: Find the task bitmaps of all resource pools to which the IP belongs for that minute based on the machine IP of the edge node, perform XOR calculations on the reported bitmap and all bitmaps of the IP resource pool one by one, and the XOR result bitmap is the list of failed tasks of the machine IP of the edge node for that minute; Step 6: Record the failed task information of the edge node service unit and perform a failed refresh operation.

2. The method for tracking task deployment from a center to an edge node according to claim 1, characterized in that: In step 2, the task ID generation mechanism adopts a self-incrementing method. The byte has 16 bits in total, the first ten bits are the current timestamp rounded to the minute value, and the last 6 bits are the self-incrementing count value. All task IDs of this minute start to increase from 1 to a maximum of 999999.

3. The method for tracking task deployment from a center to an edge node according to claim 2, characterized in that: The central service unit records the task list of each resource pool every minute with a bitmap, parses the minute timestamp and offset from the task ID, and sets the specific offset position to 1 on the bitmap corresponding to the minute.

4. The method for tracking task deployment from a center to an edge node according to claim 3, characterized in that: The edge node service unit uses a bitmap to record the task processing results every minute, parses the minute timestamp and offset from the task ID, and records 1 at the offset position of the bitmap if the task processing is successful, and records 0 if the task processing fails. After all tasks are processed in that minute, the bitmap fully records the results of all successful tasks.

5. The method for tracking task deployment from a center to an edge node according to claim 4, characterized in that: After the edge node service unit completes the task processing every minute, it generates a complete task status bitmap, compresses it using the Roaring Bitmap bitmap compression algorithm, and then reports the minute timestamp and bitmap to the central service.

6. A task deployment tracking system from center to edge node, characterized in that: The system comprises: The upstream system unit is responsible for issuing refresh URL tasks; The edge node service unit summarizes all task results, compresses them using a custom compression algorithm, and then reports them to the central service unit; The central service unit processes the refresh callback message of the edge node service unit.

7. A task deployment tracking system from center to edge node according to claim 6, characterized in that: The central service unit comprises: A basic data maintenance module, wherein the basic data maintenance module maintains customer information, domain name information, resource pool information, IP information, a mapping relationship between a resource pool and an IP, and a mapping relationship between a domain name and a resource pool; A task access module, which is responsible for receiving refresh tasks submitted externally, generating task IDs, and storing task ID information in a database; A callback processing module receives the processing result reported by the edge node service unit every minute.

8. The task deployment tracking system from center to edge node according to claim 6, characterized in that: The edge node service unit is divided into multiple resource pools according to the business, and one domain name will specify one resource pool to provide services.

9. A task deployment tracking system from center to edge node according to claim 8, characterized in that: The URL refresh request of the domain name requires all nodes in the resource pool providing the domain name service to execute cache content refresh.

10. The task deployment tracking system from center to edge node according to claim 6, characterized in that: The central service unit performs XOR calculation on the target task ID list in the callback processing module and the task access module, compares the failed task ID list of the edge node, and performs tracking processing.

Images