Same-machine-room source returning method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a same-machine-room source returning method and device, electronic equipment and a storage medium. The method comprises the following steps: under the condition that a file downloading request is received and a target file requested to be downloaded is not stored in a first server, determining the type of a first machine room; under the condition that the first machine room is a built machine room, determining the type of the first server according to the available space of the disk of the first server; when the first server is a newly added machine room server, processing the file downloading request by using a first same machine room back-to-source method; and when the first server is an old server in the machine room, processing the file downloading request by using a second same-machine-room back-to-source method. According to the same machine room back-to-source method provided by the embodiment of the invention, the back-to-source bandwidth cost can be reduced, and the back-to-source response time can be shortened.

Description

Technical Field

[0001] The present invention relates to the field of cloud service technology, and in particular to a same-computer-room back-to-source method, device, electronic device and storage medium. Background Art

[0002] Video CDN (Content Delivery Network) refers to a CDN system that can provide video services, generally including storage, synchronization, back-to-source, scheduling, video file parsing and other systems. When a client watches a video, the video streaming file download adopts the following method: after the client initiates a request, it first obtains the video file parsing list from the video file parsing server, and then downloads the files in the parsing list in sequence. When downloading each file, it is necessary to make a request to the scheduler, and download the file from the specified CDN edge node according to the result returned by the scheduler. If the file download request reaches the edge node, the file exists under the node, and the data is directly spit out to the client; if the file does not exist under the edge node, it is necessary to send a file download request to the upstream node, and then spit out the data to the client after receiving the upstream node data, and store the file locally; if the upstream node does not have this file, it is necessary to send a request to the upstream node of this upstream node. This file download method is called back-to-source.

[0003] In CDN back-to-source, there are usually multiple servers in the upstream computer room. If the downstream computer room cannot find a certain video file, it will send a request to the server in the upstream computer room. This cross-computer room back-to-source method has two disadvantages: Disadvantage 1: high bandwidth cost for back-to-source; Disadvantage 2: long response time for back-to-source. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide a method, device, electronic device and storage medium for returning to the source in the same machine room, so as to achieve fast and low bandwidth cost return to the source. The specific technical solution is as follows:

[0005] In a first aspect of the present invention, a same-machine-room back-to-source method is first provided, which is applied to a first server. The method includes:

[0006] When a file download request is received and the first server does not store the target file requested for download, determining the type of the first computer room;

[0007] In the case where the first computer room is an existing computer room, determining the type of the first server according to the available space on the disk of the first server;

[0008] When the first server is a newly added server in the computer room, the file download request is processed using a first same-computer room back-to-source method;

[0009] When the first server is an old server in the computer room, the file download request is processed using the second same-computer-room back-to-source method.

[0010] Optionally, when a file download request is received and the first server does not store the target file requested for download, the step of determining the type of the first computer room includes:

[0011] When a file download request is received and the first server does not store the target file requested for download, determining the available disk space of each second server in the first computer room, wherein each second server is each other server in the first computer room except the first server;

[0012] determining a first number of second servers having disk available space below a first predetermined percentage;

[0013] When the first number is less than a second preset percentage, determining that the first computer room is a newly built computer room;

[0014] When the first number is greater than or equal to the second preset percentage, it is determined that the first computer room is an established computer room.

[0015] Optionally, the step of processing the file download request by using the first same-machine-room back-to-source method includes:

[0016] Broadcast file query requests in the computer room where the file is located;

[0017] Determining whether a response message from a target server is received within a preset time period, wherein the target server is in the first computer room and stores the queried file;

[0018] If a response message from the target server is received, the source is returned to the target server;

[0019] If no response information is received from the target server, the message is returned to the upstream computer room.

[0020] Optionally, the step of processing the file download request by using the second same-machine-room back-to-source method includes:

[0021] Determine a hash value of a target file requested to be downloaded, and determine whether the target file is stored by the first server based on the hash value;

[0022] If so, return to the upstream computer room;

[0023] If not, search for a matching third server in the first computer room according to the hash value, and request to obtain the target file from the third server.

[0024] Optionally, the method further comprises:

[0025] Determining the availability of the first server disk;

[0026] Communicate with each second server in the first computer room to send the availability rate, and receive the availability rate of the disk sent by each second server.

[0027] Optionally, the method further comprises:

[0028] When the availability of the first server disk is less than a third preset percentage, filtering cold files stored in the disk; wherein the cold files are files that have not been downloaded within a preset time period;

[0029] The cold file is deleted from the disk, and the cache index of the cold file in the preset file filter is deleted.

[0030] Optionally, the method further comprises:

[0031] Store the target file obtained by returning to the source in the local disk;

[0032] A cache index of the target file is established in the cuckoo filter in the memory, wherein the cache index is used to indicate the file information managed in the first server.

[0033] Optionally, after the step of determining the type of the first computer room, the method further includes:

[0034] In the case where the first computer room is a newly built computer room, determining a hash value of a target file requested to be downloaded, and judging whether the target file is stored by the first server based on the hash value;

[0035] If so, return to the upstream computer room;

[0036] If not, search for a matching third server in the first computer room according to the hash value, and request to obtain the target file from the third server.

[0037] In a second aspect of the present invention, a same-machine-room back-to-source device is provided, wherein the device comprises:

[0038] A first determination module is used to determine the type of the first computer room when a file download request is received and the first server does not store the target file requested for download;

[0039] A second determination module is used to determine the type of the first server according to the available disk space of the first server when the first computer room is an existing computer room;

[0040] A first back-to-source module, configured to process the file download request using a first same-machine-room back-to-source method when the first server is a newly added machine-room server;

[0041] The second retrieval module is used to process the file download request by using a second retrieval method in the same computer room when the first server is an old server in the computer room.

[0042] Optionally, the first determination module includes: an available space determination submodule, used to determine the available disk space of each second server in the first computer room when a file download request is received and the target file requested to be downloaded is not stored in the first server, wherein the second servers are other servers in the first computer room except the first server; a quantity determination submodule, used to determine a first number of second servers whose available disk space is lower than a first preset percentage; a type determination submodule, used to determine that the first computer room is a newly built computer room when the first number is less than a second preset percentage; and to determine that the first computer room is an existing computer room when the first number is greater than or equal to the second preset percentage.

[0043] Optionally, the first retrieval module includes: a broadcast submodule, which is used to broadcast a file query request in the computer room where the first server is located when the first server is a newly added server in the computer room; a first judgment submodule, which is used to judge whether a response information of the target server is received within a preset time period, wherein the target server is in the first computer room and stores the queried file; a first execution submodule, which is used to return the source to the target server if the response information of the target server is received; and a second execution submodule, which is used to return the source to the upstream computer room if the response information of the target server is not received.

[0044] Optionally, the second back-to-source module includes: a second judgment submodule, used to determine the hash value of the target file requested to be downloaded when the first server is an old server in the computer room, and judge whether the target file is stored by the first server based on the hash value; a third execution submodule, used to back-source to the upstream computer room if so; and a fourth execution submodule, used to search for a matching third server in the first computer room according to the hash value if not, and request to obtain the target file from the third server side.

[0045] Optionally, the device also includes: an availability determination module, used to determine the availability of the first server disk; a communication module, used to communicate with each second server in the first computer room to send the availability, and receive the availability of the disk sent by each second server.

[0046] Optionally, the device also includes: a screening module, used to screen the cold files stored in the disk when the availability of the first server disk is less than a third preset percentage; wherein the cold files are files that have not been downloaded within a preset time period; and a deleting module, used to delete the cold files from the disk and delete the cache index of the cold files in the preset file filter.

[0047] Optionally, the device also includes: a storage module, used to store the target file obtained by returning to the source in a local disk; an index establishment module, used to establish a cache index of the target file in the cuckoo filter in the resident memory, wherein the cache index is used to indicate the file information managed in the first server.

[0048] Optionally, the device also includes: a third back-to-source module, which is used to determine the hash value of the target file requested to be downloaded if the first computer room is a newly built computer room after the first determination module determines the type of the first computer room, and judge whether the target file is stored by the first server based on the hash value; a policy execution module, which is used to, if so, back-source to the upstream computer room; if not, search for a matching third server in the first computer room according to the hash value, and request to obtain the target file from the third server.

[0049] In another aspect of the implementation of the present invention, there is also provided an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; the memory is used to store computer programs; the processor is used to implement any of the above-mentioned same-machine-room return-to-source methods when executing the program stored in the memory.

[0050] In another aspect of the present invention, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is executed, the computer implements any of the above-mentioned same-machine-room return-to-source methods.

[0051] In another aspect of the present invention, a computer program product including instructions is provided, which, when executed on a computer, enables the computer to implement any of the above-mentioned same-machine-room return-to-source methods.

[0052] The same-computer-room retrieval method, device, electronic device and storage medium provided by the embodiments of the present invention determine the type of the first computer room when a file download request is received and the target file requested to be downloaded is not stored in the first server. When the first computer room is an existing computer room, the type of the first server is determined according to the available disk space of the first server. When the first server is a newly added server in the computer room, the file download request is processed by the first same-computer-room retrieval method. When the first server is an old server in the computer room, the file download request is processed by the second same-computer-room retrieval method. It is possible to flexibly switch between the two same-computer-room retrieval methods according to the different types of the computer room and the first server. On the one hand, since the same-computer-room retrieval usually uses the intranet, the bandwidth cost of the retrieval can be reduced; on the other hand, since the same-computer-room retrieval has a short retrieval distance, the retrieval response time is short. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0054] Figure 1 The present invention is a flowchart of a method for returning to the same computer room in an embodiment of the present invention.

[0055] Figure 2 The present invention is a flowchart of another method for returning to the source in the same machine room according to an embodiment of the present invention.

[0056] Figure 3 The figure is a schematic diagram of the working principle of a same-machine-room backhaul system in an embodiment of the present invention.

[0057] Figure 4 The present invention is a structural block diagram of a same-machine-room source return device in an embodiment of the present invention.

[0058] Figure 5 The figure is a structural block diagram of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0060] Figure 1 The present invention is a flowchart of a method for returning to the same computer room in an embodiment of the present invention.

[0061] like Figure 1 As shown, the same machine room source return method of the embodiment of the present invention may include the following steps:

[0062] Step 101: when a file download request is received and the first server does not store the target file requested for download, determine the type of the first computer room.

[0063] The same-computer-room return-to-source method provided in the embodiment of the present invention is suitable for the scenario in which the user watches the video through the client and downloads the streaming file. Among them, the type of the computer room includes: a first type for indicating a newly built computer room, and a second type for indicating an already built computer room. When determining the type of the first computer room, it can be determined based on the disk space occupancy rate of each server in the first computer room, or it can be determined based on the construction time of the first computer room, or it can be determined based on the type label set by the computer room maintenance personnel for the first computer room, and this is not specifically limited in the embodiment of the present application.

[0064] Each server in the existing computer room has stored enough files. Different servers in the computer room usually store files according to the principle of consistent hashing. Therefore, if individual servers in the existing computer room are offline, only a few files will be rehashed and will not cause a large number of re-sources. The disk space occupancy rate of each server in the newly built computer room is low, so it is necessary to quickly preheat and store the files that it should serve. In this case, another method of returning to the source in the same computer room can be used. Therefore, in the embodiment of the present application, it is necessary to first determine the type of computer room.

[0065] Step 102: When the first computer room is an existing computer room, determine the type of the first server according to the available space on the first server disk.

[0066] The types of servers include: a third type for indicating that the server is a newly added server in the computer room, and a fourth type for indicating that the server is an old server in the computer room.

[0067] The same-machine-room retrieval method provided in the embodiment of the present invention not only takes into account the type of the machine room, but also further takes into account the type of the first server that receives the file download request, so as to provide a more efficient retrieval service. Different types of first servers use different same-machine-room retrieval methods. Among them, the same-machine-room retrieval method includes: a consistent hash retrieval method and a query retrieval method.

[0068] In an optional embodiment, after the step of determining the type of the first computer room, the method further includes: if the first computer room is a newly built computer room, determining the hash value of the target file requested to be downloaded, and judging whether the target file is stored by the first server based on the hash value; if so, returning to the upstream computer room; if not, searching for a matching third server in the first computer room based on the hash value, and requesting to obtain the target file from the third server.

[0069] In this optional embodiment, when it is determined that the first computer room is a newly built computer room, since most servers in the computer room do not have disk preheating, a consistent hashing back-to-source method is used to return the file requested to be downloaded.

[0070] Step 103, when the first server is a newly added server in the computer room, the file download request is processed using the first same-computer room back-to-source method.

[0071] The first same-machine-room tracing method may be a query tracing method, but is not limited thereto, and may also be any other appropriate same-machine-room tracing method.

[0072] Step 104, when the first server is an old server in the computer room, the file download request is processed using the second same-computer-room back-to-source method.

[0073] The second same-machine-room tracing method may be a consistent hashing tracing method, but is not limited thereto, and may also be any other appropriate same-machine-room tracing method.

[0074] The consistent hashing back-to-source method is as follows: When a server A receives a file download request, it can use consistent hashing to confirm whether the file should be served and stored by the server. If so, it can normally back-source to the upstream computer room; if not, it can request the corresponding server B in the same computer room according to the hash result. If B has the file, it will directly spit out the file to A. If it does not have the file, server B will directly back-source to the upstream computer room, and server B will send the back-to-source file to server A.

[0075] The query back-to-source method is generally used when a new server is online in the computer room. At this time, the new server A that is online usually has an empty disk and needs to be quickly preheated to store the files that it should serve. In this case, the query back-to-source method in the same computer room can be used. Every time the new server A receives a file download request, it multicasts this request to the entire computer room. Server B that receives the file download request queries whether there is such a file locally, and responds if there is. The new server A that receives the response will return the source to server B. If server A does not receive any response within the preset time, the new server A will directly return the source to the upstream computer room.

[0076] The same-computer-room retrieval method provided by the embodiment of the present invention determines the type of the first computer room when a file download request is received and the target file requested for download is not stored in the first server. When the first computer room is an existing computer room, the type of the first server is determined according to the available disk space of the first server. When the first server is a newly added server in the computer room, the first same-computer-room retrieval method is used to process the file download request; when the first server is an old server in the computer room, the second same-computer-room retrieval method is used to process the file download request. It can be realized that according to the different types of computer rooms and first servers, it is possible to flexibly switch between the two same-computer-room retrieval methods. On the one hand, since the retrieval within the same computer room usually uses the intranet, the bandwidth cost of the retrieval can be reduced; on the other hand, since the retrieval is in the same computer room, the retrieval distance is short, so the retrieval response time is short.

[0077] Figure 2 The present invention is a flowchart of another method for returning to the source in the same machine room according to an embodiment of the present invention.

[0078] like Figure 2 As shown, the same machine room source return method of the embodiment of the present invention may include the following steps:

[0079] Step 201: when a file download request is received and the target file requested for download is not stored in the first server, determine the available disk space of each second server in the first computer room.

[0080] Among them, each second server is each other server in the first computer room except the first server.

[0081] The same-machine-room retrieval method provided by the embodiment of the present invention can minimize the resource usage of querying files when retrieval is performed in the same-machine-room, and can respond to user requests more quickly. At the same time, various situations are comprehensively considered: such as building a new machine room, preheating a new server, stable operation of an old server, and other scenarios, and different same-machine-room retrieval methods can be flexibly switched.

[0082] The available space of the disk can be quantified by the disk availability. In the actual implementation process, each server in the first computer room can regularly and actively exchange disk availability information with other servers in the same computer room, so that each server can determine the status of the corresponding server based on the received disk availability. Taking the disk availability exchange between the first server and each second server in the first computer room as an example, the availability of the disk of the first server is determined; the availability is sent by communicating with each second server in the first computer room, and the availability of the disk sent by each second server is received.

[0083] Step 202: Determine the type of the first computer room based on the available disk space of each server in the first computer room.

[0084] The types of computer rooms include: newly built computer rooms and existing computer rooms, wherein newly built computer rooms can be regarded as the first type and existing computer rooms can be regarded as the second type. A method for optionally determining the type of the first computer room based on the available disk space of each server in the first computer room includes the following sub-steps:

[0085] Sub-step 1: determining a first number of second servers whose disk available space is less than a first preset percentage;

[0086] Sub-step 2: when the first number is less than a second preset percentage, determine that the first computer room is a newly built computer room; when the first number is greater than or equal to the second preset percentage, determine that the first computer room is an existing computer room.

[0087] The first preset percentage and the second preset percentage can be flexibly set by those skilled in the art, and are not specifically limited in the embodiments of the present application. For example, the first preset percentage is set to 30%, and the second preset percentage is set to 10%, that is, the available disk space of the second server in the same computer room as the first server is obtained, and the available disk space of each second server is counted. If less than 30% of the second servers have a disk available space less than 10% (that is, most of the second servers have no disk preheating), the first newly built computer room is considered to be a newly built computer room; otherwise, the first computer room is considered to be an existing computer room.

[0088] This method of quantifying the type of a computer room by using the available disk space of each server in the computer room can produce more objective and accurate results.

[0089] If the first computer room is an existing computer room, step 203 and subsequent processes are executed. If the first computer room is an existing computer room, the consistent hashing back-to-source method is used to process the file download request. The specific processing flow is described in detail below and will not be repeated here.

[0090] Step 203: When the first computer room is an existing computer room, determine the type of the first server according to the available space on the first server disk.

[0091] Among them, the types of servers include: newly added servers in the computer room and old servers in the computer room. Among them, the newly added servers in the computer room can be regarded as the third type, and the old servers in the computer room can be regarded as the fourth type.

[0092] In the actual implementation process, the size of the available disk space of the first server can be compared with the fourth preset percentage. If the available disk space of the first server is greater than the fourth preset percentage, the first server is determined to be a newly added server in the computer room. Otherwise, the first server is determined to be an old server in the computer room.

[0093] The fourth preset percentage is greater than the first preset percentage. The specific value of the fourth preset percentage can be flexibly set by those skilled in the art, and is not specifically limited in the embodiments of the present invention. For example, the fourth preset percentage can be set to 40%, 50% or 45%.

[0094] After determining the type of the first server, determine a same-machine-room back-to-source method that matches the type of the first server, and process the file download request, as detailed in steps 204 and 205. The same-machine-room back-to-source method may include but is not limited to: a consistent hash back-to-source method and a query back-to-source method.

[0095] Step 204: When the first server is a newly added server in the computer room, a query back-to-source method is used to process the file download request.

[0096] In an optional embodiment, the method of processing a file download request using a query back-to-source method may include the following sub-steps:

[0097] Sub-step 1: when the first server is a server newly added to the computer room, broadcasting a file query request in the computer room;

[0098] The query back-to-source method is generally used when a new server is online in the computer room. At this time, the new server disk needs to be quickly preheated to store the files it serves. In this case, the same computer room query back-to-source method can be used. Every time the new server receives a file download request, it multicasts the query request corresponding to the file download request to the entire computer room.

[0099] Sub-step 2: determining whether a response message from the target server is received within a preset time period;

[0100] The preset time length can be flexibly set by those skilled in the art, and there is no specific limitation on this in the embodiment of the present invention, for example, the preset time length is set to 100ms, 50ms, etc. After the first server broadcasts the file query request, each second server in the first computer room checks whether the target file is stored in the disk, and when a second server (i.e., the target server) detects that the target file exists in the disk, it sends a response message to the first server.

[0101] The target server is in the first computer room and stores the queried file, namely the target file.

[0102] It should be noted that each second server in the first computer room can confirm whether the target file is stored in the disk through the cuckoo filter and feedback the corresponding response information. If the load of the third server storing the target file in the disk is higher than the preset load, it may not respond to this query.

[0103] Sub-step 3: If a response message from the target server is received, the source is returned to the target server;

[0104] Sub-step 4: If no response information is received from the target server, return to the upstream computer room.

[0105] If no response information is received from the target server within the preset time, it means that the target file requested for download is not stored in the second servers in the same computer room. Therefore, it is necessary to send a file query request to the upstream computer room. The first server receives the target file sent by the upstream computer room, stores the target file on the local disk and feeds the target file back to the user. This process can be called upstream computer room return to the source.

[0106] Step 205: When the first server is an old server in the computer room, a consistent hashing back-to-source method is used to process the file download request.

[0107] In an optional embodiment, the method of processing a file download request using a consistent hashing back-to-source method in the same data center may include the following sub-steps:

[0108] Sub-step 1: when the first server is an old server in the computer room, determining a hash value of the target file requested for download, and judging whether the target file is stored by the first server based on the hash value;

[0109] Sub-step 2: If yes, return to the upstream computer room;

[0110] Sub-step 3: If not, search for a matching third server in the first computer room based on the hash value, and request to obtain the target file from the third server.

[0111] When requesting to obtain the target file from the third server, the first server sends a file query request to the third server in the same computer room, and the third server responds to the file query request by searching the disk for the target file and then feeding back to the first server. The third server is one of the multiple second servers in the first computer room.

[0112] The file query request includes a first identifier, and the first identifier is used to indicate that the file query request is a same-machine-room return-to-source request. In this optional embodiment, in order to prevent a return-to-source loop, the file query request needs to be added with the first identifier when returning to the same machine room: such as adding a flag to the HTTP request header field. When the server receives a file query request containing the first identifier, the server will not perform a second return to the same machine room, but will directly return to the upstream machine room.

[0113] In an embodiment of the present invention, when matching the same-computer-room retrieval method according to the type of the first server, the reason why the new server goes online using the same-computer-room query retrieval is because most of the files have been stored on the second server in the same computer room. However, due to the addition of new servers in the computer room, the number of servers in the computer room has changed, and the consistent hashing rules have also changed. According to the consistent hashing retrieval method, it is impossible to accurately find the third server that should be returned to the source. Therefore, choosing to use the query retrieval method and broadcasting the file query request to each second server in the computer room using the multicast method can improve the hit rate and quickly find the target file for preheating. When the disk of the new server is preheated to a certain extent, the consistent hashing retrieval method can be used to process the user's file download request.

[0114] In an optional embodiment, the method also includes related operations for managing the target file obtained by returning to the source, specifically including: storing the target file obtained by returning to the source in a local disk; establishing a cache index of the target file in a cuckoo filter in the memory, wherein the cache index is used to indicate the file information managed in the first server.

[0115] In order to implement file queries back to the same computer room, the server can use a key-value (KV) database such as redis or directly query whether the file exists on the disk. However, a complete KV database requires a large amount of memory cache, and directly querying files from the disk requires IO operations. It can be seen that these two query methods have serious drawbacks. In order to solve the drawbacks of the above two query methods, in the embodiments of the present invention, each server needs to index the files stored on the local disk to achieve fast query. In the embodiment of the present application, a cuckoo filter is used to cache index files. The cuckoo filter solves the problem that the Bloom filter cannot delete elements with a lower space overhead. The use of this filter can significantly reduce memory usage (1 bit can represent whether a file exists). Taking a server with a disk space of 100T as an example, the average size of a single file is 10M, and a server can store approximately 10 7 If each bucket of the cuckoo filter has 4 entries, and each entry uses a 1-byte fingerprint, then storing these files only requires 4*10 7 A memory of the order of bytes is sufficient.

[0116] This method of optionally establishing a file cache index based on a cuckoo filter can save memory space.

[0117] In an optional embodiment, the first server in the first computer room also regularly cleans up files stored in the disk, and the specific operation may include: when the availability of the disk of the first server is less than a third preset percentage, filtering cold files stored in the disk; deleting the cold files from the disk, and deleting the cache index of the cold files in the preset file filter. Wherein, the cold files are files that have not been downloaded within a preset time period.

[0118] The third preset percentage can be flexibly set by those skilled in the art, and is not specifically limited in the embodiment of the present invention, for example, it can be set to 10%, 5% or 15%, etc. In addition, each second server in the first computer room can also periodically clean up the files stored in the disk, and the specific cleaning operation is similar to the first server cleaning up the files in the disk.

[0119] This method of regularly cleaning up files stored in the disk by the server can realize the disk LRU (Least recently used) mechanism and improve the effective utilization rate of the disk.

[0120] Refer to the following Figure 3 The working principle diagram of the same-machine-room back-to-source system shown in the figure uses a specific example to illustrate the same-machine-room back-to-source method provided in the embodiment of the present application.

[0121] Figure 3 In the example, the back-to-source system in the same computer room includes two servers, Server A and Server B. In the actual implementation process, the computer room can contain multiple servers. Server A and Server B in the same computer room can not only communicate with each other, but also provide back-to-source services for users, and perform back-to-source with the upstream computer room.

[0122] like Figure 3 As shown, each server in the computer room includes the following modules:

[0123] Cuckoo filter (also known as cuckoo filter): It is a filter resident in the memory, which stores the status information of whether the file exists, such as the file cache index; as described in the above embodiment, by establishing the file cache index through the cuckoo filter, memory space can be saved. The same computer room back-to-source module: provides file query services for neighbor servers (i.e., other servers in the same computer room except the server itself) and local back-to-source modules. When receiving the query service (i.e., file download request) from the neighbor server, the same computer room back-to-source module uses the cuckoo filter to confirm whether the file exists and give a response. If the local load exceeds the preset load, the query service may not be responded to. At the same time, the local back-to-source service also uses this module to multicast the file query message, and when the same computer room back-to-source module receives the response, it will feed it back to the back-to-source service. That is, the same computer room back-to-source module is used to execute the two same computer room back-to-source methods shown in the above embodiments.

[0124] Resource monitoring module: responsible for counting the resource consumption of this server for query by the back-source module in the same computer room. The resource monitoring modules between servers can also communicate with each other to report the disk status of the local machine (such as disk availability). Specifically, UDP (User Datagram Protocol) messages can be used between servers to query files and communicate resource status, so that servers in the same computer room can easily obtain the disk status of other servers.

[0125] Back-to-source proxy module: responsible for receiving the user's file download request. If the target file corresponding to the file download request is not available locally, it will go back to the upstream computer room or the same computer room, store the data on the disk and send it to the user. When the new file is stored on the disk, the target file is registered with the Cuckoo filter.

[0126] Disk management service: implements the disk LRU mechanism, deletes cold files on the disk when the disk is almost full, and instructs the Cuckoo filter to delete the cache index of this file. Setting up the disk management service in the server can make efficient use of disk space.

[0127] When performing back-to-source in the same data center, assuming that server A receives a request from a user to download a file, the back-to-source proxy module checks whether the target file requested by the user to download exists locally on server A. If the target file does not exist locally on server A, it executes back-to-source to the upstream data center or the same data center, writes the data to the disk, and spits it out to the user. After the target file obtained by back-to-source is written to the disk, that is, after the target file is obtained, the target file is registered with the cuckoo filter (i.e. Figure 3 ), server A outputs the target file, i.e., the output data, to the user. If the user requests to download the target file from the server next time, server A can directly output the target file to the user without the need to go through the same data center back-to-source process.

[0128] Specifically, when server A executes backhaul to the upstream computer room or the same computer room, the type of computer room where service A is located is determined; when the computer room is an existing computer room, the type of server A is determined based on the available disk space of server A monitored by the resource monitoring module. When server A is a newly added server in the computer room, the query backhaul method is used to perform backhaul in the same computer room to obtain the target file. When server A is an old server in the computer room, the consistent hash backhaul method is used to perform backhaul in the same computer room to obtain the target file. It should be noted that the resource monitoring module can not only monitor the available disk space of the server where it is located, but also communicate with the resource monitoring modules of other servers in the same computer room. Figure 3 As shown in the figure, the two servers communicate through UDP messages to exchange the available disk space of the servers, so that the server can judge whether the computer room is an existing computer room or a new computer room based on its own disk space and the available disk space of other servers in the same computer room.

[0129] Taking the example that server B and server A belong to the same computer room and there are only these two servers in the computer room, when performing same-computer room back-to-source, the same-computer room back-to-source module broadcasts a file query request in the computer room where server A is located; and determines whether a response message from server B is received within a preset time period (if the target file is stored in server B, a response message will be sent to server A, otherwise it will not be sent to server A). If server A receives a response message from server B, it will back-source to server B through the same-computer room back-to-source module. When back-sourcing to server B, the target file can be sent through a UDP message; if no response message from server B is received, it will back-source to the upstream computer room.

[0130] The same-machine-room back-to-source system provided in this specific example can flexibly switch between two same-machine-room back-to-source modes according to the different machine rooms and server types. On the one hand, since the back-to-source in the same machine room usually uses the intranet, the bandwidth cost of the back-to-source can be reduced; on the other hand, since the back-to-source in the same machine room is short, the back-to-source response time is short. In addition, the cold files stored in the server can be deleted through the disk management service to make efficient use of the server's disk space. Figure 4 The present invention is a structural block diagram of a same-machine-room source return device in an embodiment of the present invention.

[0131] like Figure 4 As shown, the same machine room source return device of the embodiment of the present invention may include the following modules:

[0132] A first determination module 401 is used to determine the type of the first computer room when a file download request is received and the first server does not store the target file requested for download;

[0133] A second determination module 402 is used to determine the type of the first server according to the available disk space of the first server when the first computer room is an existing computer room;

[0134] A first back-to-source module 403, configured to process the file download request using a first same-machine-room back-to-source method when the first server is a newly added machine-room server;

[0135] The second retrieval module 404 is used to process the file download request by using a second retrieval method in the same computer room when the first server is an old server in the computer room.

[0136] Optionally, the first determination module includes: an available space determination submodule, used to determine the available disk space of each second server in the first computer room when a file download request is received and the target file requested to be downloaded is not stored in the first server, wherein the second servers are other servers in the first computer room except the first server; a quantity determination submodule, used to determine a first number of second servers whose available disk space is lower than a first preset percentage; a type determination submodule, used to determine that the first computer room is a newly built computer room when the first number is less than a second preset percentage; and to determine that the first computer room is an existing computer room when the first number is greater than or equal to the second preset percentage.

[0137] Optionally, the first retrieval module 403 includes: a broadcast submodule, which is used to broadcast a file query request in the computer room where the first server is located when the first server is a newly added server in the computer room; a first judgment submodule, which is used to judge whether a response information of the target server is received within a preset time period, wherein the target server is in the first computer room and stores the queried file; a first execution submodule, which is used to return the source to the target server if the response information of the target server is received; and a second execution submodule, which is used to return the source to the upstream computer room if the response information of the target server is not received.

[0138] Optionally, the second back-to-source module 404 includes: a second judgment submodule, used to determine the hash value of the target file requested to be downloaded when the first server is an old server in the computer room, and judge whether the target file is stored by the first server based on the hash value; a third execution submodule, used to return the source to the upstream computer room if so; and a fourth execution submodule, used to search for a matching third server in the first computer room according to the hash value if not, and request to obtain the target file from the third server side.

[0139] Optionally, the device also includes: an availability determination module, used to determine the availability of the first server disk; a communication module, used to communicate with each second server in the first computer room to send the availability, and receive the availability of the disk sent by each second server.

[0140] Optionally, the device also includes: a screening module, used to screen the cold files stored in the disk when the availability of the first server disk is less than a third preset percentage; wherein the cold files are files that have not been downloaded within a preset time period; and a deleting module, used to delete the cold files from the disk and delete the cache index of the cold files in the preset file filter.

[0141] Optionally, the device also includes: a storage module, used to store the target file obtained by returning to the source in a local disk; an index establishment module, used to establish a cache index of the target file in the cuckoo filter in the resident memory, wherein the cache index is used to indicate the file information managed in the first server.

[0142] Optionally, the device also includes: a third back-to-source module, which is used to determine the hash value of the target file requested to be downloaded after the first determination module 401 determines the type of the first computer room, if the first computer room is a newly built computer room, and to judge whether the target file is stored by the first server based on the hash value; a policy execution module, which is used to, if so, back-source to the upstream computer room; if not, search for a matching third server in the first computer room according to the hash value, and request to obtain the target file from the third server side.

[0143] The same-computer-room retrieval device provided by the embodiment of the present invention determines the type of the first computer room when a file download request is received and the target file requested to be downloaded is not stored in the first server. When the first computer room is an existing computer room, the type of the first server is determined according to the available disk space of the first server, and a same-computer-room retrieval method matching the type of the first server is determined to process the file download request. It can realize flexible switching between two same-computer-room retrieval methods according to the type of the computer room and the first server. On the one hand, since the retrieval is within the same computer room, the intranet is usually used, so the bandwidth cost of the retrieval can be reduced; on the second hand, since the retrieval is within the same computer room, the retrieval distance is short, so the retrieval response time is short.

[0144] The embodiment of the present invention further provides an electronic device, such as Figure 5 As shown, it includes a processor 501 , a communication interface 502 , a memory 503 and a communication bus 504 , wherein the processor 501 , the communication interface 502 , and the memory 503 communicate with each other via the communication bus 504 .

[0145] Memory 503, used for storing computer programs;

[0146] The processor 501 is used to execute the program stored in the memory 503, and implements the following steps:

[0147] When a file download request is received and the first server does not store the target file requested for download, determining the type of the first computer room;

[0148] In the case where the first computer room is an existing computer room, determining the type of the first server according to the available space on the disk of the first server;

[0149] When the first server is a newly added server in the computer room, the file download request is processed using a first same-computer room back-to-source method;

[0150] When the first server is an old server in the computer room, the file download request is processed using the second same-computer-room back-to-source method.

[0151] Optionally, when a file download request is received and the first server does not store the target file requested for download, the step of determining the type of the first computer room includes:

[0152] When a file download request is received and the first server does not store the target file requested for download, determining the available disk space of each second server in the first computer room, wherein each second server is each other server in the first computer room except the first server;

[0153] determining a first number of second servers having disk available space below a first predetermined percentage;

[0154] When the first number is less than a second preset percentage, determining that the first computer room is a newly built computer room;

[0155] When the first number is greater than or equal to the second preset percentage, it is determined that the first computer room is an established computer room.

[0156] Optionally, the step of processing the file download request by using the first same-machine-room back-to-source method includes:

[0157] Broadcast file query requests in the computer room where the file is located;

[0158] Determining whether a response message from a target server is received within a preset time period, wherein the target server is in the first computer room and stores the queried file;

[0159] If a response message from the target server is received, the source is returned to the target server;

[0160] If no response information is received from the target server, the message is returned to the upstream computer room.

[0161] Optionally, the step of processing the file download request by using the second same-machine-room back-to-source method includes:

[0162] Determine a hash value of a target file requested to be downloaded, and determine whether the target file is stored by the first server based on the hash value;

[0163] If so, return to the upstream computer room;

[0164] If not, search for a matching third server in the first computer room according to the hash value, and request to obtain the target file from the third server.

[0165] Optionally, the method further comprises:

[0166] Determining the availability of the first server disk;

[0167] Communicate with each second server in the first computer room to send the availability rate, and receive the availability rate of the disk sent by each second server.

[0168] Optionally, the method further comprises:

[0169] When the availability of the first server disk is less than a third preset percentage, filtering cold files stored in the disk; wherein the cold files are files that have not been downloaded within a preset time period;

[0170] The cold file is deleted from the disk, and the cache index of the cold file in the preset file filter is deleted.

[0171] Optionally, the method further comprises:

[0172] Store the target file obtained by returning to the source in the local disk;

[0173] A cache index of the target file is established in the cuckoo filter in the memory, wherein the cache index is used to indicate the file information managed in the first server.

[0174] Optionally, after the step of determining the type of the first computer room, the method further includes:

[0175] In the case where the first computer room is a newly built computer room, determining a hash value of a target file requested to be downloaded, and judging whether the target file is stored by the first server based on the hash value;

[0176] If so, return to the upstream computer room;

[0177] If not, a matching third server is searched in the first computer room according to the hash value, and a request is made to obtain the target file from the third server. The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0178] The communication interface is used for communication between the above terminal and other devices.

[0179] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0180] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0181] In another embodiment of the present invention, a computer-readable storage medium is provided, in which instructions are stored. When the computer-readable storage medium is executed on a computer, the computer implements the same-machine-room source return method described in any of the above embodiments.

[0182] In another embodiment provided by the present invention, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to implement the same-machine-room back-to-source method described in any one of the above embodiments.

[0183] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0184] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0185] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0186] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A method for returning to the source in the same computer room, applied to a first server, characterized in that: include: When a file download request is received and the first server does not store the target file requested for download, determining the type of the first computer room; In the case where the first computer room is an existing computer room, determining the type of the first server according to the available space on the disk of the first server; When the first server is a newly added server in the computer room, the file download request is processed using a first same-computer room back-to-source method; When the first server is an old server in the computer room, the file download request is processed using the second same-computer-room back-to-source method.

2. The method according to claim 1, characterized in that When a file download request is received and the first server does not store the target file requested for download, the step of determining the type of the first computer room includes: When a file download request is received and the first server does not store the target file requested for download, determining the available disk space of each second server in the first computer room, wherein each second server is each other server in the first computer room except the first server; determining a first number of second servers having disk available space below a first predetermined percentage; When the first number is less than a second preset percentage, determining that the first computer room is a newly built computer room; When the first number is greater than or equal to the second preset percentage, it is determined that the first computer room is an established computer room.

3. The method according to claim 1, characterized in that The step of processing the file download request by using the first same-machine-room back-to-source method includes: Broadcast file query requests in the computer room where the file is located; Determining whether a response message from a target server is received within a preset time period, wherein the target server is in the first computer room and stores the queried file; If a response message from the target server is received, the source is returned to the target server; If no response information is received from the target server, the message is returned to the upstream computer room.

4. The method according to claim 1, characterized in that: The step of processing the file download request by using the second same-machine-room back-to-source method includes: Determine a hash value of a target file requested to be downloaded, and determine whether the target file is stored by the first server based on the hash value; If so, return to the upstream computer room; If not, search for a matching third server in the first computer room according to the hash value, and request to obtain the target file from the third server.

5. The method according to claim 2, characterized in that: The method further comprises: Determining the availability of the first server disk; Communicate with each second server in the first computer room to send the availability rate, and receive the availability rate of the disk sent by each second server.

6. The method according to claim 1, characterized in that The method further comprises: When the availability of the first server disk is less than a third preset percentage, filtering cold files stored in the disk; wherein the cold files are files that have not been downloaded within a preset time period; The cold file is deleted from the disk, and the cache index of the cold file in the preset file filter is deleted.

7. The method according to claim 6, characterized in that The method further comprises: Store the target file obtained by returning to the source in the local disk; A cache index of the target file is established in the cuckoo filter in the memory, wherein the cache index is used to indicate the file information managed in the first server.

8. The method according to claim 1, characterized in that After the step of determining the type of the first computer room, the method further includes: In the case where the first computer room is a newly built computer room, determining a hash value of a target file requested to be downloaded, and judging whether the target file is stored by the first server based on the hash value; If so, return to the upstream computer room; If not, search for a matching third server in the first computer room according to the hash value, and request to obtain the target file from the third server.

9. A device for returning to the source in the same machine room, characterized in that: The device comprises: A first determination module is used to determine the type of the first computer room when a file download request is received and the first server does not store the target file requested for download; A second determination module is used to determine the type of the first server according to the available disk space of the first server when the first computer room is an existing computer room; A first back-to-source module, configured to process the file download request using a first same-machine-room back-to-source method when the first server is a newly added machine-room server; The second retrieval module is used to process the file download request by using a second retrieval method in the same computer room when the first server is an old server in the computer room.

10. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; The processor is used to implement the same-machine-room source return method as described in any one of claims 1-8 when executing the program stored in the memory.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the same-machine-room retrieval method as described in any one of claims 1-8 is implemented.

Images