A sharding caching method and system, electronic equipment and storage medium

Abstract

The application provides a sharding cache method, a system, an electronic device and a storage medium. The existing sharding function is partially optimized, the minimum sharding bytes of the first access positioning is 0-1, error responses of a source station to file size exceeding the limit can be avoided, the source return request can be reduced for part of the range request, and the whole system source return is reduced. The first source return IP address is recorded and applied to the subsequent source return request, the short connection can be changed into a long connection, the source return waiting time is reduced, and the download speed when not cached is improved.

Description

Technical Field

[0001] This invention belongs to the field of storage technology, specifically relating to a sharded caching method, system, electronic device, and storage medium. Background Technology

[0002] For large file storage, caching servers typically perform logical sharding, utilizing the HTTP protocol's Range feature to split requests into multiple files and store them on the local disk. This approach offers several advantages: 1. When origin server access is slow or the connection is interrupted, it ensures that as much of the downloaded data as possible is stored; without sharding, all data might be lost. 2. Sharded downloads prevent hotspots from concentrating on a single disk, which could overheat and impact the overall service of the machine. Usually, load balancing devices adjust the response between the caching server and the origin server, or there might be multiple redirects within the cloud service. The caching server follows each request until it receives the actual response content.

[0003] The cache server implements sharded storage optimization. However, the origin servers that the cache server faces vary, and their default configurations or used strategies differ. Some servers will respond with a 403 rejection for shard requests (range requests) that exceed the file size itself. Each shard may not receive a 200 response when returning to the origin server; instead, it may go through multiple 302 responses from the scheduling server before finally receiving a 200 Body response. The inability to provide sharded storage reduces the adaptability of the cache server.

[0004] The situation faced by cache servers when returning to the origin varies. The origin server or cloud provider's scheduling equipment may schedule each shard request to different devices in the cluster according to the origin server's scheduling policy. This causes each shard request to establish a new connection to obtain resources, wasting the CPU resources of the entire system, resulting in longer response time and slower download speed. Summary of the Invention

[0005] Therefore, the technical problem to be solved by the present invention is to provide a sharded caching method, system, electronic device and storage medium.

[0006] The above-mentioned technical objective of the present invention is achieved through the following technical solution:

[0007] A sharded caching method for the requesting end includes the following steps:

[0008] S1. Send the first request to the receiving end;

[0009] The first request includes a request type and / or a Range;

[0010] Where Range: bytes = 0-1;

[0011] S2. Receive a first response from the receiving end; the first response is based on the request type in the first request;

[0012] If the first response includes information generated by the second request, execute S3;

[0013] If the first response includes Range location information, execute S4;

[0014] S3. Generate a second request based on the first response and send it to the receiving end; the second request includes a request type and / or a Range.

[0015] Where Range: bytes = 2 - min{(M-1), content - length};

[0016] S4. Locate and store the complete fragment according to the Range in the first request;

[0017] S5. Receive a second response from the receiving end, the second response being based on the first request response, and store the fragmented file in the second response.

[0018] Preferably, after sending the second request to the receiving end, if the Nth request is initiated, the Range in the Nth request is: bytes: M*N-min((M*(N+1)-1), content-length), where M is the number of bytes in the slice size and N is a positive integer.

[0019] Preferably, after sending any request to the receiving end and receiving the corresponding response, a follow request is sent to the requesting end.

[0020] A fragmented caching method for the receiving end includes the following steps:

[0021] A1. Receive the first request from the requesting end;

[0022] A2. Verify the request type of the first request.

[0023] If the first request is a complete request, proceed to step A3;

[0024] If the first request is a fragmentation request, proceed to step A4;

[0025] A3. Generate a first response based on the first request and send it to the requesting end. The first response includes Range positioning information and complete data corresponding to the Range.

[0026] A4. Generate a first response based on the first request and send it to the requesting end. The first response includes second request generation information and file data of the first request.

[0027] Preferably, it also includes,

[0028] A5. Receive the Nth request from the requesting end, generate the Nth response based on the Nth request and send it to the requesting end, wherein the Nth request includes at least the N+1th request generation information;

[0029] A6. Repeat A5 until the current cached file ends.

[0030] Preferably, after sending any response to the requesting end, if no request is received from the requesting end within a preset time, the connection with the requesting end is interrupted, and the requesting end is blocked from accessing the device for a preset time.

[0031] A sharded caching system includes a requesting end and a receiving end;

[0032] The requesting end performs the following steps:

[0033] S1. Send the first request to the receiving end;

[0034] The first request includes a request type and / or a Range;

[0035] Where Range: bytes = 0-1;

[0036] S2. Receive a first response from the receiving end; the first response is based on the request type in the first request;

[0037] If the first response includes information generated by the second request, execute S3;

[0038] If the first response includes Range location information, execute S4;

[0039] S3. Generate a second request based on the first response and send it to the receiving end; the second request includes a request type and / or a Range.

[0040] Where Range: bytes = 2 - min{(M-1), content - length};

[0041] S4. Locate and store the complete fragment according to the Range in the first request;

[0042] S5. Receive a second response from the receiving end, the second response being based on the first request response, and store the fragmented file in the second response;

[0043] The receiving end performs the following steps:

[0044] A1. Receive the first request from the requesting end;

[0045] A2. Verify the request type of the first request.

[0046] If the first request is a complete request, proceed to step A3;

[0047] If the first request is a fragmentation request, proceed to step A4;

[0048] A3. Generate a first response based on the first request and send it to the requesting end. The first response includes Range positioning information and complete data corresponding to the Range.

[0049] A4. Generate a first response based on the first request and send it to the requesting end. The first response includes second request generation information and file data of the first request.

[0050] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements a segmented caching method.

[0051] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which implements a slice caching method when executed by a processor.

[0052] The above-described technical solution of the present invention has the following advantages compared with the prior art:

[0053] The existing fragmentation functionality has been partially optimized. The initial access now targets the smallest fragment size (bytes: 0-1), preventing the origin server from issuing incorrect responses for requests exceeding file size limits. Furthermore, for certain range requests, this reduces origin pull requests, lowering the overall system's origin pull latency. Recording the initial origin pull IP address and applying it to subsequent requests can shorten connections to longer connections, reducing origin pull wait time and improving download speeds when files are not cached. Attached Figure Description

[0054] Figure 1 This is a flowchart illustrating the sharding caching method provided in an embodiment of the present invention;

[0055] Figure 2 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0056] To provide a more detailed understanding of the features and technical content of the embodiments of this disclosure, the implementation of the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for illustrative purposes only and are not intended to limit the embodiments of this disclosure. In the following technical description, for ease of explanation, several details are used to provide a full understanding of the disclosed embodiments. However, one or more embodiments may still be implemented without these details. In other cases, well-known structures and systems may be simplified in their depiction to simplify the drawings.

[0057] The following description and accompanying drawings fully illustrate specific embodiments of the invention to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, procedural, and other changes. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. The scope of embodiments of the invention includes the entire scope of the claims and all available equivalents thereof. In this document, each embodiment may be referred to individually or collectively with the term "invention," which is merely for convenience and is not intended to automatically limit the scope of application to any single invention or inventive concept if more than one invention is disclosed. Relational terms such as "first" and "second" are used herein only to distinguish one entity or operation from another entity or operation, without requiring or implying any actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, or electronic device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, or electronic device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, or electronic device that includes said element. The various embodiments described herein are presented in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the methods, products, etc., disclosed in the embodiments, since they correspond to the method section disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section description.

[0058] This embodiment provides a sharded caching method for the requesting end, including the following steps:

[0059] S1. Send the first request to the receiving end;

[0060] The first request includes a request type and / or a Range;

[0061] Where Range: bytes = 0-1;

[0062] S2. Receive a first response from the receiving end; the first response is based on the request type in the first request;

[0063] If the first response includes information generated by the second request, execute S3;

[0064] If the first response includes Range location information, execute S4;

[0065] S3. Generate a second request based on the first response and send it to the receiving end; the second request includes a request type and / or a Range.

[0066] Where Range: bytes = 2 - min{(M-1), content - length};

[0067] S4. Locate and store the complete fragment according to the Range in the first request;

[0068] S5. Receive a second response from the receiving end, the second response being based on the first request response, and store the fragmented file in the second response.

[0069] Preferably, after sending the second request to the receiving end, if the Nth request is initiated, the Range in the Nth request is: bytes: M*N-min((M*(N+1)-1), content-length), where M is the number of bytes in the slice size and N is a positive integer.

[0070] Preferably, after sending any request to the receiving end and receiving the corresponding response, a follow request is sent to the requesting end.

[0071] A fragmented caching method for the receiving end includes the following steps:

[0072] A1. Receive the first request from the requesting end;

[0073] A2. Verify the request type of the first request.

[0074] If the first request is a complete request, proceed to step A3;

[0075] If the first request is a fragmentation request, proceed to step A4;

[0076] A3. Generate a first response based on the first request and send it to the requesting end. The first response includes Range positioning information and complete data corresponding to the Range.

[0077] A4. Generate a first response based on the first request and send it to the requesting end. The first response includes second request generation information and file data of the first request.

[0078] Preferably, it also includes,

[0079] A5. Receive the Nth request from the requesting end, generate the Nth response based on the Nth request and send it to the requesting end, wherein the Nth request includes at least the N+1th request generation information;

[0080] A6. Repeat A5 until the current cached file ends.

[0081] Preferably, after sending any response to the requesting end, if no request is received from the requesting end within a preset time, the connection with the requesting end is interrupted, and the requesting end is blocked from accessing the device for a preset time.

[0082] In this preferred embodiment, the CDN caching server performs a sharded origin pull. The first shard is followed by several 302 responses before finally obtaining the 206 content. The original implementation involved the second and even the Nth shards independently pulling from the origin server, following the origin server's response 1 multiple times until the content was obtained.

[0083] In this embodiment, when the first fragment response is 206, the origin IP or domain name of the response is recorded. When subsequent fragment requests are made, it is not necessary to follow up multiple times; the recorded IP address or domain name can be directly requested to quickly obtain the response content.

[0084] Originally, all fragmented requests, whether full or range-part requests, first requested the content of the first fragment, regardless of its size. The range parameter was set to bytes = 0-1, representing the minimum fragment size of 2 bytes. The origin server responded with the content of the first fragment, which also included a Content-Range parameter: 0-1 / Content_length. For example, with a 10MB file (10,485,760 bytes), the Content-Range response would be 0-1 / 10,485,760. Content_Length specifies the total file size. The CDN caching server then determined whether the original request was a full file request or a fragmented request (range-part request) and provided the client with either the full or range-part request.

[0085] This embodiment also provides a sharded caching system, including a requesting end and a receiving end;

[0086] The requesting end performs the following steps:

[0087] S1. Send the first request to the receiving end;

[0088] The first request includes a request type and / or a Range;

[0089] Where Range: bytes = 0-1;

[0090] S2. Receive a first response from the receiving end; the first response is based on the request type in the first request;

[0091] If the first response includes information generated by the second request, execute S3;

[0092] If the first response includes Range location information, execute S4;

[0093] S3. Generate a second request based on the first response and send it to the receiving end; the second request includes a request type and / or a Range.

[0094] Where Range: bytes = 2 - min{(M-1), content - length};

[0095] S4. Locate and store the complete fragment according to the Range in the first request;

[0096] S5. Receive a second response from the receiving end, the second response being based on the first request response, and store the fragmented file in the second response;

[0097] The receiving end performs the following steps:

[0098] A1. Receive the first request from the requesting end;

[0099] A2. Verify the request type of the first request.

[0100] If the first request is a complete request, proceed to step A3;

[0101] If the first request is a fragmentation request, proceed to step A4;

[0102] A3. Generate a first response based on the first request and send it to the requesting end. The first response includes Range positioning information and complete data corresponding to the Range.

[0103] A4. Generate a first response based on the first request and send it to the requesting end. The first response includes second request generation information and file data of the first request.

[0104] As shown in the figures, this disclosure provides an electronic device for segmented caching, including a processor 30 and a memory 31. Optionally, the electronic device may further include a communication interface 32 and a bus 33. The processor 30, communication interface 32, and memory 31 can communicate with each other via the bus 33. The communication interface 32 can be used for information transmission. The processor 30 can invoke logical instructions in the memory 31 to execute the segmented caching method of the above embodiments.

[0105] This disclosure provides a storage medium storing computer-executable instructions configured to execute the above-described sharding caching method.

[0106] The aforementioned storage media can be either transient computer-readable storage media or non-transitory computer-readable storage media. Non-transitory storage media include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, and can also be transient storage media.

[0107] The foregoing description and accompanying drawings fully illustrate embodiments of this disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, procedural, and other changes. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. Moreover, the terminology used in this application is for describing embodiments only and is not intended to limit the claims. As used in the description of embodiments and claims, the singular forms “a,” “an,” and “the” are intended to equally include the plural forms unless the context clearly indicates otherwise. Similarly, the term “and / or” as used in this application means including one or more of the associated listed items and all possible combinations thereof. Additionally, when used in this application, the term "comprise" and its variations "comprises" and / or "comprising" refer to the presence of stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. Without further limitations, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, or electronic device that includes said element. In this document, each embodiment may focus on the differences from other embodiments, and similar or identical parts between embodiments can be referred to mutually. For methods, products, etc., disclosed in the embodiments, if they correspond to the method section disclosed in the embodiments, the relevant parts can be referred to the description of the method section.

[0108] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this disclosure. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, units, and processes described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0109] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different blocks may also occur in a different order than disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. Each block in a block diagram and / or flowchart, and combinations of blocks in a block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

Claims

1. A sharded caching method, characterized in that, For the requesting end, the following steps are included: S1. Send the first request to the receiving end; The first request includes a request type and / or a Range; Where Range: bytes = 0-1; S2. Receive a first response from the receiving end; the first response is based on the request type in the first request; If the first response includes information generated by the second request, execute S3; If the first response includes Range location information, execute S4; S3. Generate a second request based on the first response and send it to the receiving end; the second request includes a request type and / or a Range. Where Range: bytes = 2 - min{(M-1), content - length}; S4. Locate and store the complete fragment based on the Range in the first response; S5. Receive a second response from the receiving end, the second response being a response to the second request, and store the fragmented file in the second response.

2. The sharded caching method according to claim 1, characterized in that, After sending the second request to the receiving end, if the Nth request is initiated, the Range in the Nth request is: bytes: M*N-min((M*(N+1)-1), content-length), where M is the size of the slice in bytes and N is a positive integer.

3. The sharded caching method according to claim 1, characterized in that, After sending an arbitrary request to the receiving end and receiving the corresponding response, send a follow request to the receiving end.

4. A fragmented caching method, characterized in that, For the receiving end, the following steps are included: A1. Receive the first request from the requesting end; A2. Verify the request type of the first request. If the first request is a complete request, proceed to step A3; If the first request is a fragmentation request, proceed to step A4; A3. Generate a first response based on the first request and send it to the requesting end. The first response includes Range positioning information and complete data corresponding to the Range. A4. Generate a first response based on the first request and send it to the requesting end. The first response includes second request generation information and file data of the first request.

5. The fragmented caching method according to claim 4, characterized in that, It also includes, A5. Receive the Nth request from the requesting end, generate the Nth response based on the Nth request and send it to the requesting end, wherein the Nth request includes at least the N+1th request generation information; A6. Repeat A5 until the current cached file ends.

6. The fragmented caching method according to claim 4 or 5, characterized in that, If no request is received from the requesting party after sending an arbitrary response to the requesting party within a preset time, the connection with the requesting party is terminated, and the requesting party is blocked from accessing the server for the preset time.

7. A sharded caching system, characterized in that, Includes the requesting end and the receiving end; The requesting end performs the following steps: S1. Send the first request to the receiving end; The first request includes a request type and / or a Range; Where Range: bytes = 0 1; S2. Receive a first response from the receiving end; the first response is based on the request type in the first request; If the first response includes information generated by the second request, execute S3; If the first response includes Range location information, execute S4; S3. Generate a second request based on the first response and send it to the receiving end; the second request includes a request type and / or a Range. Where Range: bytes = 2 min{(M 1), content length}; S4. Locate and store the complete fragment based on the Range in the first response; S5. Receive a second response from the receiving end, the second response being a response to a second request, and store the fragmented file in the second response; The receiving end performs the following steps: A1. Receive the first request from the requesting end; A2. Verify the request type of the first request. If the first request is a complete request, proceed to step A3; If the first request is a fragmentation request, proceed to step A4; A3. Generate a first response based on the first request and send it to the requesting end. The first response includes Range positioning information and complete data corresponding to the Range. A4. Generate a first response based on the first request and send it to the requesting end. The first response includes second request generation information and file data of the first request.

8. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 6.

Images