In-machine room source returning method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a same-machine room source returning method and device, electronic equipment and a storage medium. The method comprises: in the case that a file download request is received and a target file requested to be downloaded is not stored in a first server, determining the type of a first machine room; in the case that the first machine room is a built machine room, determining the type of the first server according to the available space of a disk of the first server; in the case that the first server is a newly added machine room server, processing the file download request by using a first same-machine room source returning method; and in the case that the first server is an old server in the machine room, processing the file download request by using a second same-machine room source returning method. The same-machine room source returning method provided by the embodiments of the present application can reduce the bandwidth cost of source returning and shorten the response time of source returning.

Description

Technical Field

[0001] This invention relates to the field of cloud service technology, and in particular to a method, apparatus, electronic device, and storage medium for same-datacenter origin-to-source connection. Background Technology

[0002] A video CDN (Content Delivery Network) refers to a CDN system that provides video services, typically including storage, synchronization, origin pull, scheduling, and video file parsing systems. When a client watches a video, the video stream file download works as follows: After the client initiates a request, it first obtains a list of parsed video files from the video file parsing server, and then downloads the files in the list sequentially. For each file download, a request is sent to the scheduler, and based on the scheduler's response, the file is downloaded from the specified CDN edge node. If the file exists at the edge node, the data is sent directly to the client; if the file does not exist at the edge node, a download request is sent to an upstream node. After receiving the data from the upstream node, the data is sent to the client, and the file is stored locally; if the upstream node does not have the file, a request is sent to its upstream node. This file download method is called origin pull.

[0003] In CDN origin pull operations, upstream data centers typically have multiple servers. If a downstream data center cannot find a certain video file, it will send a request to the upstream data center server. This cross-data center origin pull method has two drawbacks: drawback 1, high bandwidth cost; drawback 2, long origin pull response time. Summary of the Invention

[0004] The purpose of this invention is to provide a method, apparatus, electronic device, and storage medium for same-datacenter origin pullback, to achieve fast, low-bandwidth-cost origin pullback. The specific technical solution is as follows:

[0005] In a first aspect of this invention, a same-datacenter origin-pull method is provided, applied to a first server, the method comprising:

[0006] If a file download request is received and the target file to be downloaded is not stored in the first server, the type of the first computer room is determined.

[0007] If the first data center is an existing data center, the type of the first server is determined based on the available disk space of the first server;

[0008] When the first server is a newly added server in the data center, the file download request is processed using the first same data center origin return method.

[0009] When the first server is an old server in the data center, the file download request is processed using the second same-data-center origin pull method.

[0010] Optionally, the step of determining the type of the first data center where the file download request is received and the first server does not store the requested file for download includes:

[0011] If a file download request is received and the first server does not store the target file to be downloaded, the available disk space of each second server in the first data center is determined, wherein each second server is any other server in the first data center other than the first server.

[0012] Determine the first number of second servers whose available disk space is less than a first preset percentage;

[0013] If the first quantity is less than the second preset percentage, the first computer room is determined to be a newly built computer room;

[0014] If the first quantity is greater than or equal to the second preset percentage, the first computer room is determined to be an existing computer room.

[0015] Optionally, the step of processing the file download request using the first same-datacenter origin-fetching method includes:

[0016] Broadcast a file query request within the computer room.

[0017] Determine whether a response message is received from the target server within a preset time period, wherein the target server is located in the first computer room and stores the queried file;

[0018] If a response is received from the target server, a backhaul to the origin server is performed.

[0019] If no response is received from the target server, the request is sent back to the upstream data center.

[0020] Optionally, the step of processing the file download request using the second same-datacenter origin-fetching method includes:

[0021] Determine the hash value of the target file to be downloaded, and determine whether the target file is stored by the first server based on the hash value;

[0022] If so, back up to the upstream data center;

[0023] If not, search for a matching third server in the first data center based on the hash value, and request to obtain the target file from the third server.

[0024] Optionally, the method further includes:

[0025] Determine the availability of the disk on the first server;

[0026] The availability rate is communicated with each of the second servers in the first computer room and sent, and the disk availability rate is received from each of the second servers.

[0027] Optionally, the method further includes:

[0028] If the availability of the first server disk is less than a third preset percentage, cold files stored on the disk are filtered out; wherein, the cold files are files that have not been downloaded within a preset time period.

[0029] The cold file is deleted from the disk, and the cache index of the cold file in the preset file filter is also deleted.

[0030] Optionally, the method further includes:

[0031] Store the target file obtained from the source back on the local disk;

[0032] A cache index for the target file is established in the cuckoo filter in memory, wherein the cache index is used to indicate file information managed in the first server.

[0033] Optionally, after the step of determining the type of the first computer room, the method further includes:

[0034] If the first computer room is a newly built computer room, determine the hash value of the target file to be downloaded, and determine whether the target file is stored by the first server based on the hash value;

[0035] If so, back up to the upstream data center;

[0036] If not, search for a matching third server in the first data center based on the hash value, and request to obtain the target file from the third server.

[0037] In a second aspect of the present invention, a same-datacenter back-to-source device is also provided, wherein the device comprises:

[0038] The first determining module is used to determine the type of the first computer room when a file download request is received and the first server does not store the target file to be downloaded;

[0039] The second determining module is used to determine the type of the first server based on the available disk space of the first server when the first server room is an existing server room.

[0040] The first origin-fetching module is used to process the file download request using the first same-datacenter origin-fetching method when the first server is a newly added data center server.

[0041] The second origin-following module is used to process the file download request using a second origin-following method within the same data center when the first server is an older server in the data center.

[0042] Optionally, the first determining module includes: an available space determining submodule, used to determine the available disk space of each second server in the first data center when a file download request is received and the first server does not store the target file to be downloaded, wherein each second server is each other server in the first data center besides the first server; a quantity determining submodule, used to determine a first number of second servers whose available disk space is less than a first preset percentage; and a type determining submodule, used to determine that the first data center is a newly built data center when the first number is less than a second preset percentage; and to determine that the first data center is an existing data center when the first number is greater than or equal to the second preset percentage.

[0043] Optionally, the first origin-following module includes: a broadcast submodule, used to broadcast a file query request within the data center when the first server is a newly added data center server; a first judgment submodule, used to determine whether a response from the target server is received within a preset time period, wherein the target server is located in the first data center and stores the queried file; a first execution submodule, used to perform origin-following to the target server if a response from the target server is received; and a second execution submodule, used to perform origin-following to the upstream data center if no response from the target server is received.

[0044] Optionally, the second source retrieval module includes: a second judgment submodule, used to determine the hash value of the target file to be downloaded when the first server is an old server in the data center, and to determine whether the target file is stored by the first server based on the hash value; a third execution submodule, used to retrieve the target file from the upstream data center if yes; and a fourth execution submodule, used to find a matching third server in the first data center if no, and to request the target file from the third server if no.

[0045] Optionally, the apparatus further includes: an availability determination module for determining the availability of the disk of the first server; and a communication module for communicating with each of the second servers in the first data center to send the availability, and receiving the disk availability sent by each of the second servers.

[0046] Optionally, the device further includes: a filtering module, used to filter cold files stored in the disk when the availability of the first server disk is less than a third preset percentage; wherein the cold files are files that have not been downloaded within a preset time period; and a deletion module, used to delete the cold files from the disk and delete the cache index of the cold files in the preset file filter.

[0047] Optionally, the apparatus further includes: a storage module for storing the target file obtained from the source on a local disk; and an index building module for building a cache index of the target file in a cuckoo filter resident in memory, wherein the cache index is used to indicate file information managed in the first server.

[0048] Optionally, the apparatus further includes: a third origin-fetching module, configured to determine the hash value of the target file to be downloaded if the first data center is a newly built data center, after the first determining module determines the type of the first data center, and determine whether the target file is stored by the first server based on the hash value; and a policy execution module, configured to, if yes, fetch the target file from the upstream data center; if no, search for a matching third server in the first data center based on the hash value, and request to obtain the target file from the third server.

[0049] In another aspect of the present invention, an electronic device is also provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; the memory is used to store computer programs; and the processor is used to implement any of the above-described same-data center back-to-source methods when executing the programs stored in the memory.

[0050] In another aspect of the present invention, a computer-readable storage medium is also provided, wherein instructions are stored therein, which, when executed on a computer, cause the computer to implement any of the above-described same-data center back-to-source methods.

[0051] In another aspect of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, enables the computer to implement any of the above-described same-data center back-to-source methods.

[0052] The same-datacenter origin pull method, apparatus, electronic device, and storage medium provided in this invention determine the type of the first datacenter when a file download request is received and the target file to be downloaded is not stored on the first server. If the first datacenter is an existing datacenter, the type of the first server is determined based on the available disk space of the first server. When the first server is a newly added server in the datacenter, the file download request is processed using a first same-datacenter origin pull method. When the first server is an old server in the datacenter, the file download request is processed using a second same-datacenter origin pull method. This allows for flexible switching between two same-datacenter origin pull methods depending on the type of the datacenter and the first server. Firstly, since same-datacenter origin pull typically uses the intranet, the bandwidth cost of origin pull can be reduced. Secondly, since same-datacenter origin pull has a short origin pull distance, the origin pull response time is short. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0054] Figure 1 This is a flowchart illustrating the steps of a method for returning data from the same data center in an embodiment of the present invention.

[0055] Figure 2 This is a flowchart illustrating another method for same-datacenter back-to-source connection in an embodiment of the present invention.

[0056] Figure 3 This is a schematic diagram illustrating the working principle of a same-data center source return system in an embodiment of the present invention.

[0057] Figure 4 This is a structural block diagram of a data center back-to-source device according to an embodiment of the present invention.

[0058] Figure 5 This is a structural block diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0059] The technical solutions of the present invention will now be described with reference to the accompanying drawings in the embodiments of the present invention.

[0060] Figure 1 This is a flowchart illustrating the steps of a method for returning data from the same data center in an embodiment of the present invention.

[0061] like Figure 1 As shown, the same-datacenter origin-follow-up method of this invention may include the following steps:

[0062] Step 101: If a file download request is received and the target file to be downloaded is not stored in the first server, determine the type of the first data center.

[0063] The same-datacenter origin-following method provided in this embodiment of the invention is suitable for scenarios where users watch videos and download streaming files via a client. The datacenter type includes: a first type indicating a newly built datacenter, and a second type indicating an existing datacenter. The type of the first datacenter can be determined based on the disk space occupancy rate of each server in the first datacenter, the setup time of the first datacenter, or the type label set by the datacenter maintenance personnel. This embodiment does not impose specific limitations on these methods.

[0064] In established data centers, servers already store a sufficient number of files. These servers typically store files using consistent hashing principles. Therefore, if individual servers in an established data center go offline or offline, only a small number of files will be rehashed, preventing a large-scale re-origin retrieval process. However, in newly built data centers, servers have low disk space utilization, necessitating rapid preheating of files intended for other services. In this case, an alternative re-origin retrieval method within the same data center can be used. Therefore, in this embodiment, the type of data center needs to be determined first.

[0065] Step 102: If the first data center is an existing data center, determine the type of the first server based on the available disk space of the first server.

[0066] The server types include: a third type indicating that the server is a newly added server in the data center, and a fourth type indicating that the server is an old server in the data center.

[0067] The same-datacenter origin pull method provided in this invention not only considers the datacenter type but also the type of the first server receiving the file download request, thus providing a more efficient origin pull service. Different types of first servers require different same-datacenter origin pull methods. These same-datacenter origin pull methods include: consistent hashing origin pull method and query origin pull method.

[0068] In an optional embodiment, after determining the type of the first data center, the method further includes: if the first data center is a newly built data center, determining the hash value of the target file to be downloaded, and determining whether the target file is stored by the first server based on the hash value; if yes, returning to the source from the upstream data center; if no, searching for a matching third server in the first data center based on the hash value, and requesting to obtain the target file from the third server.

[0069] In this optional embodiment, when it is determined that the first data center is a newly built data center, since most servers in the data center do not have disk preheating, the consistent hashing source code method is used to perform the source code for this request to download the file.

[0070] Step 103: When the first server is a newly added server in the data center, use the first same data center origin return method to process file download requests.

[0071] The first same-datacenter origin return method can be a query origin return method, but it is not limited to this and can also be any other appropriate same-datacenter origin return method.

[0072] Step 104: When the first server is an old server in the data center, use the second same-data-center origin pull method to process file download requests.

[0073] The second same-datacenter origin return method can be a consistent hash origin return method, but it is not limited to this and can also be any other appropriate same-datacenter origin return method.

[0074] The consistent hashing origin pull method is as follows: When server A receives a file download request, it can use consistent hashing to determine whether the file should be served and stored by this server. If so, it can normally pull the file back to the upstream data center. If not, it can request the file from the corresponding server B in the same data center according to the hash result. If server B has the file, it will directly send the file to server A. If server B does not have the file, it will directly pull the file back to the upstream data center and then send the retrieved file back to server A.

[0075] The origin pull method is typically used when a new server comes online in a data center. At this time, the new server A usually has empty disks and needs to quickly warm up the files it's supposed to serve. In this case, the same-datacenter origin pull method can be used. Each time new server A receives a file download request, it multicasts the request throughout the entire data center. Server B, which receives the file download request, checks if the file exists locally; if so, it responds. New server A, receiving the response, then pulls the origin from server B. If server A does not receive any response within a preset time, it directly pulls the origin from the upstream data center.

[0076] The same-datacenter origin pull method provided in this invention determines the type of the first datacenter when a file download request is received and the target file is not stored on the first server. If the first datacenter is an existing datacenter, the type of the first server is determined based on the available disk space. When the first server is a newly added server in the datacenter, a first same-datacenter origin pull method is used to process the file download request. When the first server is an older server in the datacenter, a second same-datacenter origin pull method is used to process the file download request. This allows for flexible switching between two same-datacenter origin pull methods depending on the datacenter and the type of the first server. Firstly, since same-datacenter origin pull typically uses the intranet, the bandwidth cost of origin pull can be reduced. Secondly, since same-datacenter origin pull has a short origin pull distance, the origin pull response time is short.

[0077] Figure 2 This is a flowchart illustrating another method for same-datacenter back-to-source connection in an embodiment of the present invention.

[0078] like Figure 2 As shown, the same-datacenter origin-follow-up method of this invention may include the following steps:

[0079] Step 201: If a file download request is received and the first server does not store the target file to be downloaded, determine the available disk space of each second server in the first data center.

[0080] Each of the second servers refers to the other servers in the first data center besides the first server.

[0081] The same-datacenter origin pull method provided in this invention can minimize the resource consumption of file queries during same-datacenter origin pulls and can respond to user requests relatively quickly. It also comprehensively considers various scenarios such as newly built data centers, new server warm-up, and stable operation of old servers, flexibly switching between different same-datacenter origin pull methods.

[0082] The available space on a disk can be quantified using disk availability. In practice, each server in the first data center can periodically and proactively exchange disk availability information with other servers in the same data center, so that each server can determine its own status based on the received disk availability information. Taking the disk availability interaction between the first server and each of the second servers in the first data center as an example, the first server determines its disk availability; it communicates with each of the second servers in the first data center to send its availability information, and receives the disk availability information sent by each of the second servers.

[0083] Step 202: Determine the type of the first data center based on the available disk space of each server in the first data center.

[0084] The types of data centers include newly built data centers and existing data centers. Newly built data centers can be considered as type one, and existing data centers can be considered as type two. Optionally, determining the type of a first data center based on the available disk space of each server in the first data center includes the following sub-steps:

[0085] Sub-step 1: Determine the first number of second servers whose available disk space is less than a first preset percentage;

[0086] Sub-step 2: If the first quantity is less than the second preset percentage, determine the first computer room as a newly built computer room; if the first quantity is greater than or equal to the second preset percentage, determine the first computer room as an existing computer room.

[0087] The first preset percentage and the second preset percentage can be flexibly set by those skilled in the art, and no specific restrictions are imposed on them in the embodiments of this application. For example, the first preset percentage is set to 30%, and the second preset percentage is set to 10%, that is, the available disk space of the second servers in the same data center as the first server is obtained, and the available disk space of each second server is counted. If less than 30% of the second servers have less than 10% of their disk space available (i.e., most of the second servers have no disk warm-up), then the first newly built data center is considered to be a newly built data center; otherwise, the first data center is considered to be an existing data center.

[0088] This method of determining the type of data center by quantifying the available disk space of each server in the data center yields a more objective and accurate result.

[0089] If the first data center is an existing data center, proceed to step 203 and subsequent steps. If the first data center is an existing data center, the consistent hashing origin-based method is used to handle file download requests. The specific processing flow is described in detail below and will not be repeated here.

[0090] Step 203: If the first data center is an existing data center, determine the type of the first server based on the available disk space of the first server.

[0091] The types of servers include: newly added servers in the data center and existing servers in the data center. Newly added servers can be considered as the third type, and existing servers in the data center can be considered as the fourth type.

[0092] In actual implementation, the available disk space of the first server can be compared with the fourth preset percentage. If the available disk space of the first server is greater than the fourth preset percentage, the first server is determined to be a newly added server in the data center; otherwise, the first server is determined to be an old server in the data center.

[0093] The fourth preset percentage is greater than the first preset percentage. The specific value of the fourth preset percentage can be flexibly set by those skilled in the art, and this embodiment of the invention does not impose specific limitations on it. For example, the fourth preset percentage can be set to 40%, 50%, or 45%, etc.

[0094] After determining the type of the first server, determine the same-datacenter origin pull method that matches the type of the first server to process the file download request. See steps 204 and 205 for details. The same-datacenter origin pull method may include, but is not limited to, the consistent hash origin pull method and the query origin pull method.

[0095] Step 204: If the first server is a newly added server in the data center, the file download request is processed using the query-to-origin method.

[0096] In one optional embodiment, the method of processing file download requests using the query-to-origin method may include the following sub-steps:

[0097] Sub-step 1: If the first server is a newly added server in the data center, broadcast a file query request within the data center.

[0098] The query-to-origin method is typically used when a new server comes online in a data center. At this time, the new server's disk needs to be quickly warmed up to store the files it will be serving. In this case, the same-datacenter query-to-origin method can be used. Each time the new server receives a file download request, it multicasts the corresponding query request to the entire data center.

[0099] Sub-step 2: Determine whether a response message has been received from the target server within the preset time period;

[0100] The preset duration can be flexibly set by those skilled in the art, and no specific limitation is made in this embodiment of the invention. For example, the preset duration can be set to 100ms, 50ms, etc. After the first server broadcasts the file query request, each of the second servers in the first computer room checks whether the target file is stored on its disk. When a second server (i.e., the target server) finds that the target file exists on its disk, it sends a response message to the first server.

[0101] The target server is located in the first data center and stores the queried file, i.e., the target file.

[0102] It should be noted that each of the second servers in the first data center can use the Cuckoo Filter to check whether the target file is stored on its disk and then respond accordingly. If the load of the third server that stores the target file on its disk is higher than the preset load, it may not respond to this query.

[0103] Sub-step 3: If a response is received from the target server, then send a homing request to the target server.

[0104] Sub-step 4: If no response is received from the target server, then send a request to the upstream data center.

[0105] If no response is received from the target server within the preset time period, it means that none of the second servers in the same data center have stored the requested target file. Therefore, a file query request needs to be sent to the upstream data center. The first server receives the target file sent by the upstream data center, stores the target file on its local disk, and sends the target file back to the user. This process can be called upstream data center back-to-origin.

[0106] Step 205: If the first server is an old server in the data center, use the consistent hashing origin method to process file download requests.

[0107] In one optional embodiment, the method of using the same data center consistent hashing origin method to process file download requests may include the following sub-steps:

[0108] Sub-step 1: If the first server is an old server in the data center, determine the hash value of the target file to be downloaded, and determine whether the target file is stored by the first server based on the hash value;

[0109] Sub-step 2: If so, back up to the upstream data center;

[0110] Sub-step 3: If not, search for a matching third server in the first data center based on the hash value, and request to obtain the target file from the third server.

[0111] When a request is made to retrieve a target file from a third server, the first server sends a file query request to the third server in the same data center. The third server responds to the file query request by searching for the target file on its disk and then sends the result back to the first server. The third server is one of several second servers located in the first data center.

[0112] The file query request includes a first identifier, which indicates that the file query request is a same-datacenter origin pull request. In this optional embodiment, to prevent origin pull loops, the file query request for same-datacenter origin pulls needs to include the first identifier: such as adding a flag to the HTTP request header field. When the server receives a file query request containing the first identifier, the server will not perform a second same-datacenter origin pull, but will directly pull the file to the upstream datacenter.

[0113] In this embodiment of the invention, when matching the same-datacenter origin pull method based on the type of the first server, the reason why the same-datacenter query origin pull is used when a new server comes online is because most files are already stored on the second server in the same datacenter. However, due to the addition of a new server to the datacenter, the number of servers in the datacenter has changed, and the consistent hashing rules have also changed. The consistent hashing origin pull method cannot accurately find the third server that should be pulled back. Therefore, the query origin pull method is chosen, and file query requests are broadcast to each of the second servers in the datacenter using multicast. This improves the hit rate and quickly finds the target file for preheating. Once the new server's disk has been preheated to a certain extent, the consistent hashing origin pull method can be used to process user file download requests.

[0114] In an optional embodiment, the method further includes operations related to managing the target file obtained from the source, specifically including: storing the target file obtained from the source on a local disk; and establishing a cache index of the target file in a cuckoo filter in memory, wherein the cache index is used to indicate file information managed in the first server.

[0115] To enable file lookup from the same data center, servers can use key-value (KV) databases like Redis or directly check if a file exists on disk. However, a complete KV database requires significant memory caching, and directly querying files from disk involves I / O operations, highlighting the significant drawbacks of these two methods. To address these shortcomings, this invention indexes the files stored on the local disk of each server for faster lookup. This embodiment uses a cuckoo filter for file caching and indexing. The cuckoo filter solves the problem of Bloom filters being unable to delete elements with lower space overhead. Using this filter significantly reduces memory usage (one bit can represent the existence of a file). For example, with a server's disk space of 100TB and an average file size of 10MB, a server can store approximately 10... 7 Each file. For example, if each bucket in the Cuckoo filter has 4 entries, and each entry uses a 1-byte fingerprint, then storing these files only requires 4 * 10^6 files. 7 A memory size on the order of bytes is sufficient.

[0116] This optional method of building a file cache index based on the cuckoo filter can save memory space.

[0117] In one optional embodiment, the first server in the first data center also periodically cleans up the files stored on its disk. Specific operations may include: when the availability of the first server's disk is less than a third preset percentage, filtering for cold files stored on the disk; deleting the cold files from the disk; and deleting the cache index of cold files from the preset file filter. Cold files are files that have not been downloaded within a preset time period.

[0118] The third preset percentage can be flexibly set by those skilled in the art, and no specific limitation is made in this embodiment of the invention. For example, it can be set to 10%, 5%, or 15%. Furthermore, each of the second servers in the first data center can also periodically clean up the files stored on their disks. The specific cleaning operation is similar to the file cleaning operation performed by the first server.

[0119] This method of regularly cleaning up files stored on the disk can implement the disk LRU (Least Recently Used) mechanism, improving the effective utilization of the disk.

[0120] The following reference Figure 3 The schematic diagram of the same-datacenter source return system shown in the figure illustrates the same-datacenter source return method provided in the embodiments of this application with a specific example.

[0121] Figure 3 This example illustrates a same-datacenter origin pull system comprising two servers, Server A and Server B. In practice, a data center can contain multiple servers. Servers A and B within the same data center can not only communicate with each other but also provide origin pull services to users and facilitate origin pulls with upstream data centers.

[0122] like Figure 3 As shown, the servers in the data center include the following modules:

[0123] The Cuckoo Filter (also known as the Cuckoo Backdrop Filter) is a memory-resident filter that stores file existence status information, such as a file cache index. As described in the above embodiments, using the Cuckoo Filter to build the file cache index can save memory space. The In-Data Center Origin Module provides file query services to neighboring servers (i.e., other servers in the same data center besides this server) and the local origin module. When a query service (i.e., a file download request) is received from a neighboring server, the In-Data Center Origin Module uses the Cuckoo Filter to confirm the file's existence and provide a response. If the local machine's load exceeds a preset load, it may not respond to the query service. Simultaneously, the local origin service also uses this module to multicast file query messages. Once the In-Data Center Origin Module receives a response, it sends it back to the origin service. In other words, the In-Data Center Origin Module is used to execute the two In-Data Center Origin Module methods shown in the aforementioned embodiments.

[0124] Resource monitoring module: Responsible for statistical analysis of resource consumption on this server, providing data for querying by the origin server module within the same data center. Resource monitoring modules on different servers can also communicate with each other to report their local disk status (e.g., disk availability). Specifically, servers can use UDP (User Datagram Protocol) messages for file queries and resource information exchange, allowing servers within the same data center to easily obtain disk information from other servers.

[0125] Origin Proxy Module: This module receives file download requests from users. If the target file for the requested file is not available locally, it requests the file from an upstream or same data center, writes the data to disk, and then sends it to the user. Once a new file is written to disk, it registers the target file with the Cuckoo Filter.

[0126] Disk management service: Implements the LRU (Least Recently Used) mechanism on the disk. When the disk is nearly full, it deletes cold files on the disk and instructs the Cuckoo Filter to delete the cached index of this file. Setting up a disk management service on the server can make efficient use of disk space.

[0127] During same-datacenter origin pull, assuming server A receives a user's request to download a file, the origin pull proxy module checks if the target file exists locally on server A. If the target file does not exist locally on server A, it executes an origin pull from the upstream or same-datacenter location, writes the data to disk, and sends it to the user. After the target file is written to disk, i.e., after obtaining the target file, it is registered with the Cuckoo Filter (i.e.,...). Figure 3 As shown in the diagram (marked), server A outputs the target file, i.e., the output data, to the user. If the user requests to download the target file from the server again, server A can directly output the target file to the user without having to go through the same data center origin-back process again.

[0128] Specifically, when server A performs a backhaul to the upstream or same data center, the type of data center where server A is located is determined. If the data center is an existing data center, the type of server A is determined based on the available disk space monitored by the resource monitoring module. If server A is a newly added server in the data center, a query-based backhaul method is used to perform a backhaul within the same data center to obtain the target file. If server A is an older server in the data center, a consistent hashing backhaul method is used to perform a backhaul within the same data center to obtain the target file. It should be noted that the resource monitoring module can not only monitor the available disk space of the server itself, but also communicate with the resource monitoring modules of other servers in the same data center. Figure 3 The communication shown uses UDP messages to exchange information about the available disk space on the servers they are on. This allows the servers to determine whether the data center they are in is an established data center or a newly built data center based on their own disk space and the available disk space of other servers in the same data center.

[0129] Taking a scenario where server B and server A reside in the same data center, and only these two servers exist in the data center, the same-data-center origin pull module broadcasts a file query request within the data center of server A during the origin pull process. It then determines whether a response is received from server B within a preset timeframe. (If server B stores the target file, it will send a response to server A; otherwise, it will not.) If server A receives a response from server B, it performs the origin pull to server B through the same-data-center origin pull module. During the origin pull to server B, the target file can be sent via UDP messages. If no response is received from server B, the origin pull is then performed from the upstream data center.

[0130] This specific example provides a same-datacenter origin pull system that allows for flexible switching between two origin pull methods depending on the data center and server type. Firstly, since origin pulls within the same data center typically utilize the internal network, bandwidth costs are reduced. Secondly, the shorter origin pull distance results in shorter response times. Furthermore, the system allows for the deletion of cold files stored on the server through disk management services, ensuring efficient use of server disk space. Figure 4 This is a structural block diagram of a data center back-to-source device according to an embodiment of the present invention.

[0131] like Figure 4 As shown, the same-datacenter back-to-source device in this embodiment of the invention may include the following modules:

[0132] The first determining module 401 is used to determine the type of the first computer room when a file download request is received and the first server does not store the target file to be downloaded;

[0133] The second determining module 402 is used to determine the type of the first server based on the available disk space of the first server when the first server room is an existing server room.

[0134] The first origin-fetching module 403 is used to process the file download request using the first same-data-room origin-fetching method when the first server is a newly added data center server.

[0135] The second origin-following module 404 is used to process the file download request using a second origin-following method within the same data center when the first server is an old server in the data center.

[0136] Optionally, the first determining module includes: an available space determining submodule, used to determine the available disk space of each second server in the first data center when a file download request is received and the first server does not store the target file to be downloaded, wherein each second server is each other server in the first data center besides the first server; a quantity determining submodule, used to determine a first number of second servers whose available disk space is less than a first preset percentage; and a type determining submodule, used to determine that the first data center is a newly built data center when the first number is less than a second preset percentage; and to determine that the first data center is an existing data center when the first number is greater than or equal to the second preset percentage.

[0137] Optionally, the first origin-following module 403 includes: a broadcast submodule, used to broadcast a file query request within the data center when the first server is a newly added data center server; a first judgment submodule, used to determine whether a response information from a target server is received within a preset time period, wherein the target server is located in the first data center and stores the queried file; a first execution submodule, used to perform origin-following to the target server if a response information from the target server is received; and a second execution submodule, used to perform origin-following to the upstream data center if no response information from the target server is received.

[0138] Optionally, the second source retrieval module 404 includes: a second judgment submodule, used to determine the hash value of the target file to be downloaded when the first server is an old server in the data center, and to determine whether the target file is stored by the first server based on the hash value; a third execution submodule, used to retrieve the target file from the upstream data center if yes; and a fourth execution submodule, used to search for a matching third server in the first data center based on the hash value if no, and to request the target file from the third server.

[0139] Optionally, the apparatus further includes: an availability determination module for determining the availability of the disk of the first server; and a communication module for communicating with each of the second servers in the first data center to send the availability, and receiving the disk availability sent by each of the second servers.

[0140] Optionally, the device further includes: a filtering module, used to filter cold files stored in the disk when the availability of the first server disk is less than a third preset percentage; wherein the cold files are files that have not been downloaded within a preset time period; and a deletion module, used to delete the cold files from the disk and delete the cache index of the cold files in the preset file filter.

[0141] Optionally, the apparatus further includes: a storage module for storing the target file obtained from the source on a local disk; and an index building module for building a cache index of the target file in a cuckoo filter resident in memory, wherein the cache index is used to indicate file information managed in the first server.

[0142] Optionally, the apparatus further includes: a third origin-fetching module, configured to, after the first determining module 401 determines the type of the first data center, determine the hash value of the target file to be downloaded if the first data center is a newly built data center, and determine whether the target file is stored by the first server based on the hash value; and a policy execution module, configured to, if yes, fetch the target file from the upstream data center; if no, search for a matching third server in the first data center based on the hash value, and request to obtain the target file from the third server.

[0143] The same-datacenter origin pull device provided in this embodiment of the invention determines the type of the first datacenter when a file download request is received and the target file to be downloaded is not stored on the first server. If the first datacenter is an established datacenter, the type of the first server is determined based on the available disk space of the first server. The same-datacenter origin pull method matching the type of the first server is then determined to process the file download request. This allows for flexible switching between two same-datacenter origin pull methods depending on the type of datacenter and the first server. Firstly, since origin pull within the same datacenter typically uses the intranet, the bandwidth cost of origin pull can be reduced. Secondly, since origin pull within the same datacenter has a short origin pull distance, the origin pull response time is short.

[0144] This invention also provides an electronic device, such as... Figure 5 As shown, it includes a processor 501, a communication interface 502, a memory 503, and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504.

[0145] Memory 503 is used to store computer programs;

[0146] When processor 501 executes the program stored in memory 503, it performs the following steps:

[0147] If a file download request is received and the target file to be downloaded is not stored in the first server, the type of the first computer room is determined.

[0148] If the first data center is an existing data center, the type of the first server is determined based on the available disk space of the first server;

[0149] When the first server is a newly added server in the data center, the file download request is processed using the first same data center origin return method.

[0150] When the first server is an old server in the data center, the file download request is processed using the second same-data-center origin pull method.

[0151] Optionally, the step of determining the type of the first data center where the file download request is received and the first server does not store the requested file for download includes:

[0152] If a file download request is received and the first server does not store the target file to be downloaded, the available disk space of each second server in the first data center is determined, wherein each second server is any other server in the first data center other than the first server.

[0153] Determine the first number of second servers whose available disk space is less than a first preset percentage;

[0154] If the first quantity is less than the second preset percentage, the first computer room is determined to be a newly built computer room;

[0155] If the first quantity is greater than or equal to the second preset percentage, the first computer room is determined to be an existing computer room.

[0156] Optionally, the step of processing the file download request using the first same-datacenter origin-fetching method includes:

[0157] Broadcast a file query request within the computer room.

[0158] Determine whether a response message is received from the target server within a preset time period, wherein the target server is located in the first computer room and stores the queried file;

[0159] If a response is received from the target server, a backhaul to the origin server is performed.

[0160] If no response is received from the target server, the request is sent back to the upstream data center.

[0161] Optionally, the step of processing the file download request using the second same-datacenter origin-fetching method includes:

[0162] Determine the hash value of the target file to be downloaded, and determine whether the target file is stored by the first server based on the hash value;

[0163] If so, back up to the upstream data center;

[0164] If not, search for a matching third server in the first data center based on the hash value, and request to obtain the target file from the third server.

[0165] Optionally, the method further includes:

[0166] Determine the availability of the disk on the first server;

[0167] The availability rate is communicated with each of the second servers in the first computer room and sent, and the disk availability rate is received from each of the second servers.

[0168] Optionally, the method further includes:

[0169] If the availability of the first server disk is less than a third preset percentage, cold files stored on the disk are filtered out; wherein, the cold files are files that have not been downloaded within a preset time period.

[0170] The cold file is deleted from the disk, and the cache index of the cold file in the preset file filter is also deleted.

[0171] Optionally, the method further includes:

[0172] Store the target file obtained from the source back on the local disk;

[0173] A cache index for the target file is established in the cuckoo filter in memory, wherein the cache index is used to indicate file information managed in the first server.

[0174] Optionally, after the step of determining the type of the first computer room, the method further includes:

[0175] If the first computer room is a newly built computer room, determine the hash value of the target file to be downloaded, and determine whether the target file is stored by the first server based on the hash value;

[0176] If so, back up to the upstream data center;

[0177] If not, search for a matching third server in the first computer room based on the hash value and request the target file from the third server. The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the figure, but this does not mean that there is only one bus or one type of bus.

[0178] The communication interface is used for communication between the aforementioned terminal and other devices.

[0179] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0180] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0181] In another embodiment of the present invention, a computer-readable storage medium is also provided, which stores instructions that, when executed on a computer, cause the computer to implement any of the same data center back-to-source methods described in the above embodiments.

[0182] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, enables the computer to implement any of the same data center back-to-source methods described in the above embodiments.

[0183] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0184] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0185] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0186] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A method for same-datacenter origin pullback, applied to a first server, characterized in that, include: If a file download request is received and the target file to be downloaded is not stored in the first server, the type of the first computer room is determined. If the first data center is an existing data center, the type of the first server is determined based on the available disk space of the first server; When the first server is a newly added server in the data center, a file query request is broadcast within the data center. It is determined whether a response is received from the target server within a preset time period, wherein the target server is located in the first data center and stores the queried file. If a response is received from the target server, a backhaul is performed to the target server. If no response is received from the target server, a backhaul is performed to the upstream data center. When the first server is an old server in the data center, the hash value of the target file to be downloaded is determined, and the target file is determined based on the hash value to see if it is stored by the first server; if yes, the origin is retrieved from the upstream data center; if no, a matching third server is found in the first data center based on the hash value, and the target file is requested to be obtained from the third server.

2. The method according to claim 1, characterized in that, The step of determining the type of the first data center when a file download request is received and the first server does not store the requested file for download includes: If a file download request is received and the first server does not store the target file to be downloaded, the available disk space of each second server in the first data center is determined, wherein each second server is any other server in the first data center other than the first server. Determine the first number of second servers whose available disk space is less than a first preset percentage; If the first quantity is less than the second preset percentage, the first computer room is determined to be a newly built computer room; If the first quantity is greater than or equal to the second preset percentage, the first computer room is determined to be an existing computer room.

3. The method according to claim 2, characterized in that, The method further includes: Determine the availability of the disk on the first server; The availability rate is communicated with each of the second servers in the first computer room and sent, and the disk availability rate is received from each of the second servers.

4. The method according to claim 1, characterized in that, The method further includes: If the availability of the first server disk is less than a third preset percentage, cold files stored on the disk are filtered out; wherein, the cold files are files that have not been downloaded within a preset time period. The cold file is deleted from the disk, and the cache index of the cold file in the preset file filter is also deleted.

5. The method according to claim 4, characterized in that, The method further includes: Store the target file obtained from the source back on the local disk; A cache index for the target file is established in the cuckoo filter in memory, wherein the cache index is used to indicate file information managed in the first server.

6. The method according to claim 1, characterized in that, Following the step of determining the type of the first computer room, the method further includes: If the first computer room is a newly built computer room, determine the hash value of the target file to be downloaded, and determine whether the target file is stored by the first server based on the hash value; If so, back up to the upstream data center; If not, search for a matching third server in the first data center based on the hash value, and request to obtain the target file from the third server.

7. A source return device in the same computer room, characterized in that, The device includes: The first determining module is used to determine the type of the first computer room when a file download request is received and the target file to be downloaded is not stored in the first server; The second determining module is used to determine the type of the first server based on the available disk space of the first server when the first server room is an existing server room. The first origin-following module is used to broadcast a file query request within the data center when the first server is a newly added data center server; determine whether a response is received from the target server within a preset time period, wherein the target server is located in the first data center and stores the queried file; if a response is received from the target server, then origin-following is performed to the target server; if no response is received from the target server, then origin-following is performed to the upstream data center. The second origin-fetching module is used to determine the hash value of the target file to be downloaded when the first server is an old server in the data center, and to determine whether the target file is stored by the first server based on the hash value; if yes, it fetches the target file from the upstream data center; if no, it searches for a matching third server in the first data center based on the hash value and requests to obtain the target file from the third server.

8. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; The processor, when executing a program stored in memory, implements the same-datacenter back-to-source method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the same-datacenter back-to-origin method as described in any one of claims 1-6.

Images