A method and apparatus for refreshing a CDN shard

Abstract

The application provides a CDN refresh fragment method and device, belonging to the technical field of content distribution network. The method comprises the following steps: S1: receiving a target fragment refresh request, deleting the index slice corresponding to the fragment; S2: receiving a target fragment service request, inquiring whether the index slice corresponding to the fragment exists based on service logic, and setting a mark bit on the fragment if the index slice does not exist; S3: initiating a back-to-source request based on back-to-source logic, obtaining the size of a complete file, and assembling the URL of all fragments according to the size of the complete file; S4: obtaining the meta information of all fragments according to the URL of all fragments, and performing a refresh operation according to the meta information of the fragment if the mark bit is set on the fragment. The application avoids the disk reading operation in the original method, and does not have any impact on the disk load; when the refresh request is high concurrency, the disk is no longer a bottleneck, and the normal service request is not affected.

Description

Technical Field

[0001] This invention relates to the field of content delivery network technology, and more specifically, to a method and apparatus for refreshing CDN shards. Background Technology

[0002] Content Delivery Networks (CDNs) work by using nodes at various levels to cache resources from remote origin servers to nodes closer to users. Users can then access resources from the nearest node, thereby reducing the load on the origin server and improving access speed.

[0003] Currently, when processing large files such as videos and downloads, especially those containing range requests of varying sizes, CDN providers generally use a sharding approach to minimize origin pull traffic and improve hit rates while correctly responding to customer requests: sharded origin pull, sharded storage, and sharded refresh.

[0004] In practice, the typical process of shard refresh is as follows: upon receiving a refresh request, the contents of the index shard are located and read using the URL provided by the refresh task. This allows the URLs and storage information of all shards to be obtained directly or calculated. The refresh operation is then executed to refresh the shards. The index shard is a file containing complete file storage information and the URLs of each shard file. Due to the large number and size of index shard files, they are unsuitable for storage in memory and are generally stored on disk. In practice, it has been found that with high concurrency of refresh requests, the shard refresh method described above increases disk load due to the operation of reading the index shards, which can severely impact normal business requests. Summary of the Invention

[0005] To address the aforementioned issues, this application provides a method and apparatus for refreshing CDN shards.

[0006] Firstly, this application provides a method for real-time CDN shard refresh, including the following steps: S1: Receive the target shard refresh request and delete the index shard corresponding to the shard; S2: Receive the target shard service request, query whether the index shard corresponding to the shard exists based on the business logic, and set a flag bit on the shard if the index shard does not exist. S3: Initiate a request to the origin server based on the origin server logic, obtain the size of the complete file, and assemble the URLs of all fragments according to the size of the complete file; S4: Retrieve the metadata of each shard based on its URL. If a flag is set on the shard, perform a refresh operation based on the shard's metadata.

[0007] Preferably, step S1 specifically includes: Receive a refresh request for the target shard, obtain the corresponding URL in the refresh task based on the received refresh request, find the index shard from the disk based on the URL, and delete the index shard.

[0008] Preferably, step S3 specifically includes: responding to the business request when no index shard is found, taking the action of pulling from the origin server, receiving the response from the origin server, obtaining the total file length, and calculating and assembling the URLs of all shards based on the total file length and the shard size.

[0009] Preferably, the index slice is deleted via memory operations.

[0010] Preferably, the index slice is queried through memory operations.

[0011] Secondly, embodiments of this application provide a CDN refresh sharding apparatus, including... The delete index module is used to receive the target shard refresh request and delete the index shard corresponding to the shard. The query index module is used to receive the target shard business request, query whether the index shard corresponding to the shard exists based on the business logic, and set a flag bit on the shard if the index shard does not exist. The fragment assembly module is used to initiate a back-to-origin request based on the back-to-origin logic, obtain the size of the complete file, and assemble the URLs of all fragments according to the size of the complete file; The refresh execution module is used to obtain the metadata of each shard based on the URL of each shard. If a flag is set on the shard, the refresh operation is performed based on the metadata of the shard.

[0012] Specifically, the index deletion module includes: receiving a refresh request for the target shard, obtaining the corresponding URL in the refresh task based on the received refresh request, finding the index shard from the disk based on the URL, and deleting the index shard.

[0013] Specifically, the shard assembly module includes: responding to business requests when no index shard is found, pulling from the source, receiving the response from the source server, obtaining the total file length, and calculating and assembling the URLs of all shards based on the total file length and shard size.

[0014] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method provided as in the first aspect or any possible implementation of the first aspect.

[0015] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method provided as in the first aspect or any possible implementation thereof.

[0016] The beneficial effects of this invention are: it avoids the disk read operation in the original method and will not have any impact on disk load; when refresh requests are highly concurrent, the disk will no longer become a bottleneck and will not affect normal business requests. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating a CDN sharding refresh method provided in an embodiment of this application; Figure 2 A schematic diagram of a CDN refresh sharding device provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application; Figure 4 A schematic diagram illustrating the process of using the method of this application to refresh shards in a CDN shard refresh method provided in an embodiment of this application; Figure 5 A schematic diagram of the test environment when using the method of this application to refresh shards in a CDN shard refresh method provided in an embodiment of this application; Figure 6 A schematic diagram illustrating the process of using existing technology to refresh shards in a CDN sharding refresh method provided in this application embodiment; Figure 7 This is a diagram illustrating the disk's behavior when performing shard refresh using existing technology in a CDN shard refresh method provided in this embodiment of the application. Detailed Implementation

[0019] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0020] In the following description, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The following description provides multiple embodiments of this application, which can be substituted or combined with each other. Therefore, this application can also be considered to include all possible combinations of the same and / or different embodiments described. Thus, if one embodiment includes features A, B, and C, and another embodiment includes features B and D, then this application should also be considered to include embodiments containing one or more other possible combinations of A, B, C, and D, even if such embodiments are not explicitly described in the following text.

[0021] The following description provides examples and does not limit the scope, applicability, or examples set forth in the claims. Changes may be made to the function and arrangement of the described elements without departing from the scope of this application. Various processes or components may be appropriately omitted, substituted, or added to the examples. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Furthermore, features described with respect to some examples may be combined into other examples.

[0022] Please see Figure 1 , 4 5, 6, 7. Figure 1 This is a flowchart illustrating a CDN shard refresh method provided in an embodiment of this application. Figure 4 This is a flowchart illustrating the CDN shard refresh method provided in this application embodiment, illustrating the shard refresh process using the method described in this application. Figure 5 This diagram illustrates the test environment when using the CDN shard refresh method provided in this embodiment of the application for shard refresh. Figure 6 This is a schematic diagram illustrating the process of using existing technology to refresh shards in a CDN sharding refresh method provided in an embodiment of this application. Figure 7 This image illustrates the disk's behavior when performing shard refresh using existing technology in a CDN shard refresh method provided in this embodiment. The method includes the following steps: S1: Receive the target shard refresh request and delete the index shard corresponding to the shard; S2: Receive the target shard service request, query whether the index shard corresponding to the shard exists based on the business logic, and set a flag bit on the shard if the index shard does not exist. S3: Initiate a request to the origin server based on the origin server logic, obtain the size of the complete file, and assemble the URLs of all fragments according to the size of the complete file; S4: Retrieve the metadata of each shard based on its URL. If a flag is set on the shard, perform a refresh operation based on the shard's metadata.

[0023] In the embodiments of this application, the inventors discovered in practical work that the existing sharded refresh method, when the concurrency of refresh requests is high, increases the disk load due to the operation of reading index shards, which can seriously affect normal business requests. For example... Figure 7 This is the performance of a disk during a high-concurrency refresh request stress test. It can be seen that the number of reads increases significantly and the load rises significantly.

[0024] Given the performance bottlenecks of existing sharded refresh methods, in order to meet the high concurrency requirements of sharded refresh tasks and reduce the impact of refresh operations on disk load, a new sharded refresh method is proposed. The aim is to meet the high concurrency requirements of sharded refresh tasks while reducing the impact of refresh operations on disk load, thus better solving the sharded refresh problem.

[0025] In one possible implementation, step S1 specifically includes: Receive a refresh request for the target shard, obtain the corresponding URL in the refresh task based on the received refresh request, find the index shard from the disk based on the URL, and delete the index shard.

[0026] In one possible implementation, step S3 specifically includes: responding to a business request when no index shard is found, performing a back-to-origin retrieval, receiving the origin server response, obtaining the total file length, and calculating and assembling the URLs of all shards based on the total file length and shard size.

[0027] In one possible implementation, the index slice is deleted via memory operations.

[0028] In one possible implementation, the index slice is queried via memory operations.

[0029] In this embodiment of the application, a method for efficiently refreshing CDN shards is described in detail below: 1) Upon receiving a refresh request, the system no longer checks whether the index slice exists or reads the file content of the index slice. Instead, it directly deletes the memory index of the index slice. Deleting the index is a memory operation and will not affect the disk.

[0030] 2) When a business request is received again, check if the index piece file exists. If the index piece does not exist, set a special flag to indicate that the file is not in the cache or has been refreshed. Checking if the index piece exists is also a memory operation and will not affect the disk. 3) When responding to a business request without an index slice, it is necessary to fetch the file from the origin server. After receiving the response from the origin server, the Content-Length of the file, i.e., the total file length, can be determined. 4) Given that the total file length and fragment size are already determined, the URLs of all fragments can be calculated and assembled using certain rules. 5) After obtaining the metadata of the fragments, if a special flag has been set previously, a refresh operation will be performed. In summary, the main idea of ​​this invention is that upon receiving a shard refresh request, the index shard is not checked for existence, nor is its content read; instead, the index shard is directly deleted. When the file is requested by the client again, the following operations are performed using the business logic and origin logic: checking for the existence of the index shard, setting a special flag, obtaining the remaining shard metadata, and performing a shard refresh operation.

[0031] In the refresh method of the present invention, if the index piece does not exist in step 1, an additional memory operation will be added, but the original refresh purpose can be achieved while avoiding the disk read operation in the original method.

[0032] The CDN shard refresh method of this invention splits the shard refresh process in existing methods into two parts. When a shard refresh request arrives, the URL in the refresh task is obtained. Without querying, regardless of whether it exists, the index shard is directly deleted, and the index can no longer be queried. When a real business request arrives, in order to respond to the customer, the index is queried according to normal business logic. If a refresh operation has already been performed, it will not be queried, meaning that the file has not been requested or refreshed. In this method, all requests are assumed to have been refreshed, and a flag is set to indicate this state. According to normal business logic, the file is requested from the origin server in shards. The size of the complete file can be obtained from the origin server response. Based on the complete file size and the known shard size, the URLs of each shard are assembled through simple calculations. Based on the URLs of each shard, the shard refresh operation is performed.

[0033] In the above embodiments, deleting and querying the index are both memory operations, while other logics such as the back-to-origin request reuse the logic of the business request itself, avoiding the disk read operation in the original method, and the refresh operation will no longer increase the disk load.

[0034] After successfully executing the above embodiments, the stress test performance was significantly improved compared to before. For example... Figure 5 This is disk monitoring during high-concurrency load testing of shard refresh requests in a test environment. `sda` represents the system and log disks (which can be ignored), and other data disks did not experience any load change due to shard refresh. This application only triggers memory deletion operations when shard refresh is performed. By assuming a refresh operation has already been executed and setting a flag, the actual refresh operation is attached to the normal business logic of the business request, thus avoiding disk reads for shard refresh.

[0035] The following will be combined with the appendix Figure 2 This application provides a detailed description of a CDN refresh sharding apparatus according to embodiments. It should be noted that the appendix... Figure 2 The apparatus shown is a CDN shard refresh device used to perform the functions described in this application. Figure 1 The methods shown in the embodiments are illustrated for ease of explanation, showing only the parts relevant to the embodiments of this application. For specific technical details not disclosed, please refer to this application. Figure 1 The example shown.

[0036] Please see Figure 2 , Figure 2 This is a schematic diagram of a CDN refresh sharding device provided in an embodiment of this application. Figure 2 As shown, the device includes The index deletion module 201 is used to receive the target shard refresh request and delete the index shard corresponding to the shard. The query index module 202 is used to receive the target shard business request, query whether the index shard corresponding to the shard exists based on the business logic, and set a flag bit on the shard if the index shard does not exist. The fragment assembly module 203 is used to initiate a back-to-origin request based on the back-to-origin logic, obtain the size of the complete file, and assemble the URLs of all fragments according to the size of the complete file; The refresh execution module 204 is used to obtain the metadata of all shards according to the URL of all shards. If a flag is set on the shard, the refresh operation is performed according to the metadata of the shard.

[0037] In one possible implementation, the index deletion module 201 includes: receiving a refresh request for a target shard, obtaining the corresponding URL in the refresh task based on the received refresh request, finding the index shard from the disk based on the URL, and deleting the index shard.

[0038] In one possible implementation, the fragment assembly module 203 includes: responding to a business request when no index fragment is found, performing a back-to-source fetch, receiving a response from the origin server, obtaining the total file length, and calculating and assembling the URLs of all fragments based on the total file length and fragment size.

[0039] In this embodiment, the shard refresh process in the existing method is split into two parts. The refresh task only triggers the deletion of index shards, while the business request triggers the actual shard refresh operation. If the business request cannot find the index shard, it is assumed that the file has previously undergone a shard refresh operation, and a flag is set to indicate this state. The deletion and query mentioned in this application are all memory operations, replacing the disk operations in the existing method.

[0040] Those skilled in the art will clearly understand that the technical solutions of the embodiments of this application can be implemented by means of software and / or hardware. In this specification, "unit", "module" and "part" refer to software and / or hardware that can independently complete or cooperate with other components to complete a specific function, wherein the hardware may be, for example, a field-programmable gate array (FPGA), an integrated circuit (IC), etc.

[0041] Each processing unit and / or module in the embodiments of this application can be implemented by an analog circuit that implements the functions described in the embodiments of this application, or by software that executes the functions described in the embodiments of this application.

[0042] See Figure 3 It shows a schematic diagram of the structure of an electronic device according to an embodiment of this application, which can be used to implement... Figure 1 The method in the illustrated embodiment. (As shown) Figure 3 As shown, the electronic device 300 may include: at least one central processing unit 301, at least one network interface 304, user interface 303, memory 305, and at least one communication bus 302.

[0043] The communication bus 302 is used to enable communication between these components.

[0044] The user interface 303 may include a display screen and a camera. Optionally, the user interface 303 may also include a standard wired interface and a wireless interface.

[0045] The network interface 304 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0046] The central processing unit 301 may include one or more processing cores. The central processing unit 301 connects to various parts within the electronic device 300 using various interfaces and lines. It executes various functions of the terminal 300 and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 305, and by calling data stored in the memory 305. Optionally, the central processing unit 301 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The central processing unit 301 may integrate one or more of the following: a central processing unit (CPU), a graphics processing unit (GPU), and a modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display; and the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the central processing unit 301.

[0047] The memory 305 may include random access memory (RAM) or read-only memory. Optionally, the memory 305 may include a non-transitory computer-readable storage medium. The memory 305 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 305 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 305 may also be at least one storage device located remotely from the aforementioned central processing unit 301. Figure 3 As shown, the memory 305, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and program instructions.

[0048] exist Figure 3In the illustrated electronic device 300, the user interface 303 is mainly used to provide an input interface for the user and to obtain the user's input data; while the central processing unit 301 can be used to call an application program for CDN refresh sharding stored in the memory 305, and specifically perform the following operations: S1: Receive the target shard refresh request and delete the index shard corresponding to the shard; S2: Receive the target shard service request, query whether the index shard corresponding to the shard exists based on the business logic, and set a flag bit on the shard if the index shard does not exist. S3: Initiate a request to the origin server based on the origin server logic, obtain the size of the complete file, and assemble the URLs of all fragments according to the size of the complete file; S4: Retrieve the metadata of each shard based on its URL. If a flag is set on the shard, perform a refresh operation based on the shard's metadata.

[0049] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method. The computer-readable storage medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, DVDs, CD-ROMs, microdrives, as well as magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic cards or optical cards, nanosystems (including molecular memory ICs), or any type of medium or device suitable for storing instructions and / or data.

[0050] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0051] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0052] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some service interface; the indirect coupling or communication connection between devices or units may be electrical or other forms.

[0053] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0054] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0055] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0056] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0057] The foregoing description is merely an exemplary embodiment of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Those skilled in the art will readily conceive of embodiments of this disclosure upon considering the specification and practicing the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described herein. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

Claims

1. A method for refreshing CDN shards, characterized in that, Includes the following steps: S1: Receive the target shard refresh request and delete the index shard corresponding to the shard; S2: Receive the target shard service request, query whether the index shard corresponding to the shard exists based on the business logic, and set a flag bit on the shard if the index shard does not exist. S3: Initiate a request to the origin server based on the origin server logic, obtain the size of the complete file, and assemble the URLs of all fragments according to the size of the complete file; S4: Retrieve the metadata of each shard based on its URL. If a flag is set on the shard, perform a refresh operation based on the shard's metadata. Step S3 specifically includes: responding to business requests when no index shard is found, pulling from the origin server, receiving the origin server response, obtaining the total file length, and calculating and assembling the URLs of all shards based on the total file length and shard size; Step S1 specifically includes: Receive a refresh request for the target shard, obtain the corresponding URL in the refresh task based on the received refresh request, find the index shard from the disk based on the URL, and delete the index shard. Delete index slices via memory operations; Query index slices through memory operations.

2. A CDN refresh sharding device, characterized in that: include The delete index module is used to receive the target shard refresh request and delete the index shard corresponding to the shard. The query index module is used to receive the target shard business request, query whether the index shard corresponding to the shard exists based on the business logic, and set a flag bit on the shard if the index shard does not exist. The fragment assembly module is used to initiate a back-to-origin request based on the back-to-origin logic, obtain the size of the complete file, and assemble the URLs of all fragments according to the size of the complete file; The refresh execution module is used to obtain the metadata of each shard according to the URL of each shard. If a flag is set on the shard, the refresh operation is performed according to the metadata of the shard. The shard assembly module includes: responding to business requests when no index shard is found, pulling from the source, receiving the response from the source server, obtaining the total file length, and calculating and assembling the URLs of all shards based on the total file length and shard size; The index deletion module includes: receiving a refresh request for the target fragment, obtaining the corresponding URL in the refresh task according to the received refresh request, finding the index fragment from the disk according to the URL, and deleting the index fragment. Delete index slices via memory operations; Query index slices through memory operations.

3. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in claim 1.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in claim 1.

Images