A collaborative block prefetching method based on P2P network

Abstract

The present disclosure provides a collaborative block prefetching method based on a P2P network, which divides the nodes in the P2P network into root nodes and proxy nodes. The root node is responsible for pulling data blocks from a remote warehouse, and the proxy node can only pull data blocks from the root node. At the same time, the root node actively predicts the data blocks required by the proxy node and pushes them to the proxy node, thereby realizing collaborative block prefetching of the root node and the proxy node. The present disclosure reduces the cold start delay of the virtualized instance on the proxy node, accelerates the startup and agile deployment of the virtualized instance by collaboratively prefetching the virtualized instance data blocks, quickly responds to the user's delay requirements, and reduces the overhead caused by large-scale deployment of virtualized instances.

Description

Technical Field

[0001] The present disclosure belongs to the technical field of computer data storage, and in particular, relates to a collaborative block pre-fetching method based on a P2P network. Background Art

[0002] With the widespread application of cloud computing technology, real application scenarios increasingly require flexible and fast deployment of application clusters. As the most important core of cloud computing, virtualization technology has gradually formed two frameworks suitable for modern hardware structures and networks in the course of development in recent years: virtual machines and containers. Virtual machines start the client operating system on the virtualized hardware resources at the Hypervisor layer, provide a complete operating system environment for the startup of applications, and implement resource isolation at the operating system level; containers are a lightweight virtualization technology composed of applications and their dependent packages. Resource isolation and resource allocation are implemented through Namespace and Cgroup provided by Linux. The host operating system kernel is shared between containers to implement process-level resource isolation. Compared with virtual machines, containers are smaller in size. And because they share the same operating system kernel and do not need to start the entire operating system, container applications generally start faster than virtual machines with less performance loss. Compared with lightweight virtualized containers, virtual machines can provide strong isolation. In scenarios with higher security requirements, applications need to be deployed using virtual machines.

[0003] In real application scenarios, as business develops, the application startup response speed is becoming more and more a key factor in the application experience. Although containers start faster than virtual machines, they can achieve more flexible and agile application deployment through elastic scaling, load balancing, and other methods. However, when creating or updating large container clusters, the startup of containers is still slow, and the response speed cannot fully meet the business requirements of timely response. In addition, in some environments with higher security requirements, containers cannot achieve strong resource isolation, and it is often necessary to start containers on the basis of virtual machines to achieve relative resource isolation. At this time, the bottleneck of application startup speed lies in the startup speed of the virtual machine operating system.

[0004] According to the analysis of relevant literature, an average of 80% of the time when the container is started is spent on downloading and decompressing images from remote repositories, and the startup time fluctuates greatly. Some studies have pointed out that during the container startup process, only about 6.4% of the data in the downloaded and decompressed container image is actually used, and the download and decompression of the remaining data takes up most of the deployment time, but has no effect in the application startup process. In commercial cloud computing scenarios, two feasible solutions have been proposed to solve this problem: speeding up the download process through p2p downloads to try to solve the download speed problem of large-scale cluster creation, and trying to streamline the application image to effectively reduce the startup time and space occupancy, but the current streamlined image operation is not automatic and universal enough, and cannot meet the user's dynamic loading of files for persistence, and it is difficult to support the user's temporary operation needs. To this end, it is necessary to develop a collaborative block prefetching method based on P2P network. Summary of the invention

[0005] The purpose of the present disclosure is to provide a collaborative block pre-fetching method based on a P2P network to overcome the defects of the prior art.

[0006] In order to achieve the above objectives, the technical solutions adopted in this disclosure are as follows:

[0007] A collaborative block prefetching method based on a P2P network comprises the following steps:

[0008] S1, divide the nodes in the P2P network into root nodes and proxy nodes, the proxy node starts the P2P service and connects to the root node, the root node starts the P2P service, initializes the cache pool and block request, and enters steps S2 and S3;

[0009] S2, the root node initializes the cache pool in the memory, the cache pool stores the data block cache in a key-value manner, and then proceeds to step S4;

[0010] S3, the root node initializes a block request record linked list, which includes a bidirectional linked list to store data block request records, each node in the bidirectional linked list stores data in a key-value manner, and then proceeds to step S5;

[0011] S4, the proxy node sends a data block request to the root node. The root node receives the data block request and determines whether the data block is in the cache pool. If so, it proceeds to step S6, otherwise it proceeds to step S7;

[0012] S5, the proxy node sends a data block request to the root node. The root node receives the data block request and determines whether the data block is in the block request record linked list. If so, it proceeds to step S8, otherwise it proceeds to step S9;

[0013] S6, the root node sends the data blocks in the cache pool to the proxy node and proceeds to step S4;

[0014] S7, the root node sends the data block request to the remote warehouse, receives the data block sent by the remote warehouse, inserts the data block into the cache pool, inserts the data block request record into the end of the block request record linked list, inserts the data request record metadata into the hash table, and then sends the data block to the proxy server, and goes to step S4;

[0015] S8, find the position of the hit record in the block request record linked list according to the key-value pair in the hash table, actively push the data block in the record file to the proxy server, and update the hit node to the head of the linked list, and then go to step S4;

[0016] S9. The root node inserts the requested data block record into the block request record linked list, and checks whether the length of the block request record linked list exceeds a threshold. If the length exceeds the threshold, the root node deletes the last record in the linked list according to the access frequency, and then proceeds to step S5.

[0017] Furthermore, in step S2, the key is composed of the data block cache file storage directory, the data block request file address, and the data block offset address in the file, and the value is composed of the key, file size, file descriptor, and read-write lock.

[0018] Furthermore, in step S3, the key is the data request file path, which includes the data block cache file storage directory, the data block request file address, and the data block offset address in the file, and the value is the key of the cache pool.

[0019] Furthermore, in step S3, a key-value hash table is initialized to quickly access each node in the linked list, where the key in the hash table is the data request file path, and the value is a pointer to the corresponding node in the linked list.

[0020] Furthermore, in the step S1, the nodes communicate with each other using the HTTP protocol, and the HTTP request of the proxy node is processed by the root node.

[0021] Compared with the prior art, the advantages of the present invention are as follows: the present invention divides the nodes in the P2P network into two layers of root nodes and proxy nodes. The root nodes of the first layer are responsible for pulling data blocks from the remote warehouse, and the proxy nodes can only pull data blocks from the root nodes. At the same time, the root nodes and the proxy nodes have a cache pool for caching the received data blocks, and the root nodes also have a block request record linked list for storing block request records. The hierarchical deployment of the root nodes and the proxy nodes reduces the pressure of the remote warehouse, accelerates the flow of data blocks in the P2P network, and makes the root nodes the transit and sender of the data blocks, making full use of the resources of each node. The present invention uses a collaborative block pre-fetching method in the root nodes and the proxy nodes, that is, after the proxy node sends a data block request to the root node, the root node determines whether it is in the cache pool and the block request record linked list according to the request, and pushes the data blocks requested by the proxy node and the data blocks predicted by the block request record linked list to the proxy node, giving play to the initiative of the data sender, that is, the root node, saving the application deployment time on the proxy node, and accelerating the flow of valid data blocks in the P2P network. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings illustrate exemplary embodiments of the present disclosure and together with the description serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0023] Figure 1 It is a network topology diagram of the collaborative block prefetching method based on P2P network disclosed in the present invention.

[0024] Figure 2 It is a schematic diagram of the collaborative block prefetching process in the present disclosure. DETAILED DESCRIPTION

[0025] The present disclosure is further described in detail below in conjunction with the accompanying drawings and implementations. It is understood that the specific implementations described herein are only used to explain the relevant content, rather than to limit the present disclosure. It should also be noted that, for ease of description, only the parts related to the present disclosure are shown in the accompanying drawings.

[0026] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure can be combined with each other. The technical solution of the present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0027] Unless otherwise specified, the exemplary embodiments / embodiments shown will be understood as providing exemplary features of various details of some ways in which the technical concept of the present disclosure can be implemented in practice. Therefore, unless otherwise specified, the features of the various embodiments / embodiments can be combined, separated, interchanged and / or rearranged without departing from the technical concept of the present disclosure.

[0028] The use of cross-hatching and / or shading in the accompanying drawings is generally used to make the boundaries between adjacent components clear. As such, unless otherwise specified, the presence or absence of cross-hatching or shading does not convey or indicate any preference or requirement for the specific materials, material properties, dimensions, proportions, commonalities between the components shown, and / or any other characteristics, attributes, properties, etc. of the components. In addition, in the accompanying drawings, the sizes and relative sizes of the components may be exaggerated for clarity and / or descriptive purposes. When the exemplary embodiments can be implemented differently, the specific process sequence can be performed in a different order than described. For example, two successively described processes can be performed substantially simultaneously or in an order opposite to the described order. In addition, the same figure numbers represent the same components.

[0029] When a component is referred to as being "on" or "over," "connected to," or "coupled to" another component, the component may be directly on, directly connected to, or directly coupled to the other component, or intervening components may be present. However, when a component is referred to as being "directly on," "directly connected to," or "directly coupled to" another component, there are no intervening components. For this purpose, the term "connected" may refer to a physical connection, an electrical connection, etc., with or without intervening components.

[0030] For descriptive purposes, the present disclosure may use spatially relative terms such as "under," "beneath," "under," "down," "over," "upper," "above," "higher," and "side (e.g., in a "sidewall")," to describe the relationship of one component to another (other) component as shown in the accompanying drawings. The spatially relative terms are intended to encompass different orientations of the device in use, operation, and / or manufacture in addition to the orientation depicted in the accompanying drawings. For example, if the device in the accompanying drawings is turned over, components described as "under" or "beneath" other components or features would subsequently be positioned "over" the other components or features. Thus, the exemplary term "under" can encompass both the "above" and "below" orientations. Furthermore, the device may be otherwise oriented (e.g., rotated 90 degrees or at other orientations), and the spatially relative descriptors used herein should be interpreted accordingly.

[0031] The terms used here are for the purpose of describing specific embodiments, and are not intended to be restrictive. As used here, unless the context clearly indicates otherwise, the singular forms "one (kind, person)" and "said (the)" are also intended to include plural forms. In addition, when the terms "comprise" and / or "include" and their variations are used in this specification, it is explained that there are stated features, integral bodies, steps, operations, parts, assemblies and / or their groups, but it is not excluded that there are or add one or more other features, integral bodies, steps, operations, parts, assemblies and / or their groups. It should also be noted that, as used here, the terms "substantially", "approximately" and other similar terms are used as approximate terms and not as degree terms, so that they are used to explain the inherent deviations of the measured values, calculated values ​​and / or the values ​​provided that will be recognized by those of ordinary skill in the art.

[0032] See also Figure 1 As shown, according to the network topology diagram of the present disclosure, the following operations are performed:

[0033] According to the node situation in the P2P network, the nodes are divided into root nodes and proxy nodes. The root node can directly pull data blocks from the remote warehouse, and the proxy node can only obtain data blocks from the root node. The nodes communicate using the HTTP protocol, and all HTTP requests of the proxy nodes must be processed by the root node.

[0034] The proxy node starts the P2P service, sends the data block request required by the virtualized instance deployed on the node to the root node, and receives the data blocks from the root node. These data blocks are divided into data blocks requested by the proxy node and data blocks actively pushed by the root node. After receiving the data blocks, the proxy node stores them in its own files.

[0035] The root node starts the P2P service, pulls data blocks from the remote warehouse, and receives data block requests from the proxy node. If the request hits the cache pool on the root node, the data block in the cache pool is returned to the proxy node. Otherwise, the root node requests the data block required by the proxy node from the remote warehouse. If the request hits the block request record linked list on the root node, the data block hit by the linked list is actively pushed to the proxy node, otherwise it is not actively pushed.

[0036] The root node stores cached data blocks in the form of files. At the same time, the cache pool in the memory stores the data block cache in a key-value manner, where the key consists of the data block cache file storage directory, the data block request file address, and the data block offset address in the file, and the value consists of the key, file size, file descriptor, and read-write lock.

[0037] The root node stores the block request records in a bidirectional linked list. Each node in the linked list stores data in a key-value format, where the key is the data request file path and the value is the key of the cache pool, i.e., the data block cache file storage directory, the data block request file address, and the data block offset address in the file. At the same time, a key-value hash table is maintained to quickly access each node in the linked list. In the hash table, the key is the data request file path and the value is a pointer to the corresponding node in the linked list. The bidirectional linked list uses the least recently used algorithm, i.e., the LRU algorithm, to change the block request record.

[0038] See also Figure 2 As shown, this embodiment discloses a collaborative block prefetching method based on a P2P network, comprising the following steps:

[0039] Step S1, divide the nodes in the P2P network into root nodes and proxy nodes, the proxy nodes start the P2P service and connect to the root nodes, the root nodes start the P2P service, initialize the cache pool and block request, and enter steps S2 and S3.

[0040] Step S2: The root node initializes the cache pool in the memory. The cache pool stores the data block cache in a key-value manner, where the key is composed of the data block cache file storage directory, the data block request file address, and the data block offset address in the file, and the value is composed of the key, file size, file descriptor, and read-write lock. After the cache pool is initialized, it waits for the data request to arrive and proceeds to step S4.

[0041] Step S3, the root node initializes the block request record linked list, which includes a bidirectional linked list to store data block request records. Each node in the bidirectional linked list stores data in a key-value manner, where the key is the data request file path, and the value is the key of the cache pool, that is, the data block cache file storage directory, the data block request file address, and the data block offset address in the file. At the same time, a key-value hash table is initialized to quickly access each node in the linked list. In the hash table, the key is the data request file path, and the value is a pointer to the corresponding node in the linked list. After the block request record linked list is initialized, it waits for the data request to arrive and proceeds to step S5.

[0042] Step S4: The proxy node sends a data block request to the root node. The root node receives the data block request and determines whether the data block is in the cache pool. If so, it proceeds to step S6; otherwise, it proceeds to step S7.

[0043] Step S5: The proxy node sends a data block request to the root node. The root node receives the data block request and determines whether the data block is in the block request record linked list. If so, it proceeds to step S8; otherwise, it proceeds to step S9.

[0044] Step S6: The root node sends the data blocks in the cache pool to the proxy node and proceeds to step S4.

[0045] Step S7, the root node sends the data block request to the remote warehouse, receives the data block sent by the remote warehouse, inserts the data block into the cache pool, inserts the data block request record into the end of the block request record linked list, inserts the data request record metadata into the hash table, and then sends the data block to the proxy server, and proceeds to step S4.

[0046] Step S8, find the position of the hit record in the block request record linked list according to the key-value pair in the hash table, actively push the data block in the record file to the proxy server, and update the hit node to the head of the linked list, and then go to step S4.

[0047] Step S9: The root node inserts the requested data block record into the block request record linked list, and checks whether the length of the block request record linked list exceeds a threshold. If it exceeds the threshold, the root node deletes the last record in the record linked list according to the access frequency, and then proceeds to step S5.

[0048] The present disclosure fully divides the P2P network nodes into root nodes and proxy nodes, giving full play to the roles and resources of different nodes, wherein the root node can directly pull data blocks from the remote warehouse, and the proxy node can only obtain data blocks from the root node. The nodes communicate using the HTTP protocol, and all HTTP requests of the proxy nodes must be processed by the root node.

[0049] The present disclosure deploys the data structures required for collaborative block prefetching on the root node, namely, the cache pool and the block request record linked list. The cache pool stores the data block cache in a key-value manner, where the key is composed of the data block cache file storage directory, the data block request file address, and the data block offset address in the file, and the value is composed of the key, file size, file descriptor, and read-write lock. The block request record is stored in a bidirectional linked list, and data is stored in a key-value manner in each node in the linked list, where the key is the data request file path, and the value is the key of the cache pool, namely, the data block cache file storage directory, the data block request file address, and the data block offset address in the file.

[0050] At the same time, a key-value hash table is maintained to quickly access each node in the linked list. The key in the hash table is the data request file path, and the value is a pointer to the corresponding node in the linked list. The bidirectional linked list uses the least recently used algorithm, i.e., the LRU algorithm, to change the block request record.

[0051] The present invention is based on a cache pool and a linked list of block request records. The root node and the proxy node can perform collaborative block pre-fetching operations. The root node responds to the request of the proxy node, and predicts the data blocks that the proxy node will need later and actively pushes them to the root node. The collaborative block pre-fetching mechanism requires mutual cooperation between the root node and the proxy node, and is based on the data structure on the root node.

[0052] The present invention reduces the cold start delay of virtualized instances on proxy nodes, accelerates the startup and agile deployment of virtualized instances by collaboratively pre-fetching virtualized instance data blocks, quickly responds to user delay requirements, and reduces the overhead caused by large-scale deployment of virtualized instances.

[0053] In the description of this specification, the description with reference to the terms "one embodiment / method", "some embodiments / methods", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment / method or example are included in at least one embodiment / method or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment / method or example. Moreover, the specific features, structures, materials or characteristics described may be combined in a suitable manner in any one or more embodiments / methods or examples. In addition, those skilled in the art may combine and combine the different embodiments / methods or examples described in this specification and the features of the different embodiments / methods or examples, unless they are contradictory.

[0054] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, "plurality" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0055] Those skilled in the art should understand that the above embodiments are only for the purpose of clearly illustrating the present disclosure, and are not intended to limit the scope of the present disclosure. For those skilled in the art, other changes or modifications may be made based on the above disclosure, and these changes or modifications are still within the scope of the present disclosure.

Claims

1. A collaborative block prefetching method based on a P2P network, characterized in that: The following steps are involved: S1, divide the nodes in the P2P network into root nodes and proxy nodes, the proxy node starts the P2P service and connects to the root node, the root node starts the P2P service, initializes the cache pool and block request, and enters steps S2 and S3; S2, the root node initializes the cache pool in the memory, the cache pool stores the data block cache in a key-value manner, and then proceeds to step S4; S3, the root node initializes a block request record linked list, which includes a bidirectional linked list to store data block request records, each node in the bidirectional linked list stores data in a key-value manner, and then proceeds to step S5; S4, the proxy node sends a data block request to the root node. The root node receives the data block request and determines whether the data block is in the cache pool. If so, it proceeds to step S6, otherwise it proceeds to step S7; S5, the proxy node sends a data block request to the root node. The root node receives the data block request and determines whether the data block is in the block request record linked list. If so, it proceeds to step S8, otherwise it proceeds to step S9; S6, the root node sends the data blocks in the cache pool to the proxy node and proceeds to step S4; S7, the root node sends the data block request to the remote warehouse, receives the data block sent by the remote warehouse, inserts the data block into the cache pool, inserts the data block request record into the end of the block request record linked list, inserts the data request record metadata into the hash table, and then sends the data block to the proxy server, and goes to step S4; S8, find the position of the hit record in the block request record linked list according to the key-value pair in the hash table, actively push the data block in the record file to the proxy server, and update the hit node to the head of the linked list, and then go to step S4; S9, the root node inserts the request data block record into the block request record linked list, and checks whether the length of the block request record linked list exceeds the threshold. If it exceeds the threshold, the root node deletes the last record in the record linked list according to the access frequency, and then proceeds to step S5; In step S2, the key is composed of the data block cache file storage directory, the data block request file address, and the data block offset address in the file, and the value is composed of the key, file size, file descriptor, and read-write lock; In step S3, a key-value hash table is initialized to quickly access each node in the linked list, where the key in the hash table is the data request file path, and the value is a pointer to the corresponding node in the linked list.

2. The collaborative block prefetching method based on P2P network according to claim 1, characterized in that: In step S3, key is the data request file path, which includes the data block cache file storage directory, the data block request file address, and the data block offset address in the file, and value is the key of the cache pool.

3. The collaborative block prefetching method based on P2P network according to claim 1, characterized in that: In step S1, the nodes communicate with each other using the HTTP protocol, and the HTTP request of the proxy node is processed by the root node.

Images