A mirror storage scheduling method and system based on resource heat

Abstract

The application relates to the field of cloud computing, and discloses a mirror image storage scheduling method and system based on resource heat. The method is applied to a public cloud platform, the resource pool of the public cloud platform comprises a plurality of computing nodes; the method comprises the following steps: after a target mirror image file is created, a corresponding number of computing nodes are selected from the resource pool based on a first preset value, and the computing nodes are used as storage nodes of the target mirror image file; one copy of the target mirror image file is stored in each storage node; wherein the target mirror image file is stored in each storage node in the form of a mirror image fragment; in the process of providing downloading of the target mirror image file, a downloading heat value of the target mirror image file is obtained according to a preset time period; and based on the downloading heat value, the number of copies of the target mirror image file stored in the resource pool is expanded or reduced. The method can improve the efficiency of downloading of mirror image files of a cloud platform and save the construction cost of the cloud platform.

Description

Technical Field

[0001] This application relates to the field of cloud computing, and in particular to a mirrored storage scheduling method and system based on resource popularity. Background Technology

[0002] In public cloud platforms, image services are a fundamental product feature. The cloud platform service provides image lifecycle management capabilities, allowing users to flexibly apply for elastic cloud servers and physical machines using public, private, or shared images. Furthermore, users can create private images using existing cloud servers or external image files to achieve business migration to the cloud. Through the image service, users can upload private images to object storage and download them locally, or specify an image to create a cloud server and import files from object storage to create an image. Additionally, users can share images for other users to download and use. During the transmission, download, and storage of image files, the upload and download speeds directly impact the user experience of the cloud platform.

[0003] Public cloud platforms have a large user base. Within the cloud platform's resource pool, each user has a certain quota of private images, meaning they can create and store private images in the resource pool. To meet users' image storage and download needs, the space required to store image files in the resource pool is very large. This typically involves deploying dedicated high-capacity storage servers, resulting in high cloud platform setup costs. To ensure a good user experience, while providing large-capacity storage, the download speed of image files must also be considered. However, in scenarios where multiple users concurrently download image files, the bandwidth limitations of the storage server lead to low download speeds, severely impacting the user experience.

[0004] Therefore, how to improve the efficiency of multi-user concurrent download of image files, cope with high-concurrency download demands, and enhance user experience are the problems that need to be solved. Summary of the Invention

[0005] In view of this, this application aims to propose a mirror storage scheduling method and system based on resource popularity, so as to improve the efficiency of concurrent download of image files by users and enhance the user experience of public cloud platforms.

[0006] To achieve the above objectives, the technical solution of this application is as follows:

[0007] The first aspect of this application provides a mirrored storage scheduling method based on resource popularity. The method is applied to a public cloud platform, where the resource pool of the public cloud platform includes multiple computing nodes. The method includes:

[0008] After the target image file is created, a corresponding number of computing nodes are selected from the resource pool based on a first preset value to serve as storage nodes for the target image file; a copy of the target image file is stored in each storage node; wherein, the target image file is stored in each storage node in the form of image shards;

[0009] During the download process of the target image file, the download popularity value of the target image file is obtained according to a preset time period; based on the download popularity value, the number of copies of the target image file stored in the resource pool is expanded or reduced.

[0010] Optionally, a corresponding number of computing nodes are selected from the resource pool based on a first preset value as storage nodes for the target image file, including:

[0011] Obtain the current remaining disk capacity of each computing node in the resource pool, and sort the computing nodes from largest to smallest according to the remaining disk capacity to obtain the sorting result;

[0012] Based on the first preset value, select the top-ranked computing nodes in the sorting results as the storage nodes for the target image file.

[0013] Optionally, each compute node includes multiple data disks; storing one copy of the target image file in each storage node includes:

[0014] The target image file is split into multiple image fragments according to a preset fragment size;

[0015] For each storage node, all image fragments of the target image file are distributed and stored on various data disks of the storage node.

[0016] Optionally, the download popularity value of the target image file is obtained according to a preset time period, including:

[0017] According to a preset time period, obtain information about all storage nodes that store the target image file;

[0018] Based on the information of all storage nodes, the image download logs of each storage node are obtained and parsed to obtain the download volume of the target image file within the time period, and the download volume is used as the download popularity value of the target image file.

[0019] Optionally, based on the download popularity value, the number of copies of the target image file stored in the resource pool may be increased or decreased, including:

[0020] Obtain the download popularity value within the previous time period, and compare the download popularity value with a popularity threshold; if the download popularity value is greater than the popularity threshold, determine that the target image file is highly popular; if the download popularity value is less than or equal to the popularity threshold, determine that the target image file is low popular.

[0021] When the target image file is highly popular, a target number of copies not less than a second preset value is set as the expected number of copies of the target image file to be stored within the current time period; the second preset value is determined based on the maximum network throughput rate of the resource pool, the maximum transmission rate of the computing node, and the maximum read / write rate of a single data disk; the second preset value is greater than the first preset value.

[0022] When the target image file is of low popularity, the target number of copies of the first preset value is set as the expected number of copies of the target image file to be stored in the current time period.

[0023] Based on the expected number of copies of the target image file to be stored and the actual number of copies stored within the current time period, the number of copies of the target image file stored in the resource pool is increased or decreased.

[0024] Optionally, if the download popularity value is greater than the popularity threshold, the method further includes:

[0025] Compare the target number of portions with a third preset value;

[0026] If the target number of copies is greater than the third preset value, the third preset value will be used as the expected number of copies of the target image file to be stored within the current time period.

[0027] Optionally, based on the expected and actual number of copies of the target image file stored in the resource pool within the current time period, the number of copies of the target image file stored can be increased or decreased, including:

[0028] Calculate the difference between the expected number of copies of the target image file and the actual number of copies stored within the current time period;

[0029] If the difference is positive, the difference is taken as the number of copies to be expanded; based on the number of copies to be expanded, a corresponding number of computing nodes are selected from the resource pool as new storage nodes, and the target image file is stored in the new storage nodes;

[0030] If the difference is negative, the absolute value of the difference is taken as the number of copies to be reduced; based on the number of copies to be reduced, a corresponding number of storage nodes are selected and the target image file in the storage nodes is deleted.

[0031] Optionally, storing the target image file in the new storage node includes:

[0032] Establish point-to-point communication links between all storage nodes currently storing the target image file and all new storage nodes;

[0033] Based on the point-to-point communication link, the target image file is transmitted to all new storage nodes through all storage nodes currently storing the target image file, so that each new storage node can receive different image fragments of the target image file in parallel.

[0034] Optionally, the process of providing the target image file for download also includes:

[0035] When any computing node receives a download request for the target image file sent by the client, it searches for information on all computing nodes in the resource pool that store the target image file based on the information of the target image file.

[0036] Based on the information of each computing node that stores the target image file, a point-to-point communication link is established between the client and each computing node, so that the client can download different image fragments of the target image file in parallel from each computing node.

[0037] According to a second aspect of the embodiments of this application, a mirrored storage scheduling system based on resource popularity is provided. The system includes: a cloud platform base, a peer-to-peer server, and a management platform.

[0038] The cloud platform base includes multiple computing nodes as provided in the method of the first aspect of the present application, and each computing node has a data disk with disk space for storing target image files.

[0039] The peer-to-peer server is used to collect statistics on the information of the image files stored in each computing node; and to provide information on the corresponding computing nodes for establishing peer-to-peer communication links in the resource pool.

[0040] The management platform is used to execute the steps in the method provided in the first aspect of the embodiments of this application, to manage the computing nodes and image files in the resource pool, to specify the storage nodes for each newly created image file, and to dynamically schedule the number of copies of each image file stored in the resource pool according to the download popularity value of each image file.

[0041] The image storage scheduling method based on resource popularity provided in this application involves selecting multiple computing nodes (a first preset value) from the resource pool as storage nodes after the target image file is created, and storing the target image file in image fragments on each storage node. During the cloud platform's provision of the target image file download service, the download popularity value of the target image file is obtained at certain time intervals, and the number of copies of the target image file stored in the resource pool is dynamically scheduled based on the download popularity value.

[0042] The image storage scheduling method based on resource popularity provided in this application does not use dedicated large-capacity storage servers to store image files. Instead, it stores user-created target image files on compute nodes within a resource pool. It utilizes the idle storage space within compute nodes to store image file fragments, eliminating the need for additional storage servers and saving on cloud platform deployment costs. When storing target image files, multiple redundant copies are stored on different compute nodes, allowing users to request downloads from different nodes. Compared to providing image file downloads through dedicated large-capacity storage servers, this method uses multiple compute nodes to store multiple target image files, storing them as image fragments on the compute nodes. This allows different parts of the image file to be downloaded in parallel, improving download speed, reducing the load on individual compute nodes, and enhancing the overall performance of the cloud platform. Attached Figure Description

[0043] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a flowchart of a mirrored storage scheduling method based on resource popularity proposed in an embodiment of this application;

[0045] Figure 2 This is a flowchart illustrating the scheduling of the number of copies of a target image file in one embodiment of this application;

[0046] Figure 3 This is a schematic diagram of a mirrored storage scheduling system based on resource popularity proposed in an embodiment of this application. Detailed Implementation

[0047] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0048] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0049] In the various embodiments of this application, it should be understood that the sequence number of each process described below does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0050] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects as detailed in this application.

[0051] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0052] The image service on the public cloud platform provides official images, private images, and shared images. Among them, public images are common standard operating system images, visible to all users, including the operating system and pre-installed public applications.

[0053] Private images contain images of the operating system or business data, pre-installed public applications, and the user's private applications, and are only visible to the individual user. Private images include system disk images, data disk images, ISO images, and full machine images. System disk images contain the operating system and application software required for the user's business operations and can be used to create cloud servers and migrate user businesses to the cloud. Data disk images contain only the user's business data and can be used to create cloud disks to migrate user business data to the cloud. ISO images are private images created by registering the ISO file of an external image to the cloud platform; they are special images and can only be distributed as temporary cloud servers. Full machine images, also called complete images, contain the operating system, application software, and business data required for the user's business operations. Full machine images are created based on differential backups, resulting in higher creation efficiency compared to system disk images and data disk images of the same disk capacity. Users can share their created private images with other users for download and use; these are known as shared images.

[0054] This application provides a method for dynamically scheduling the storage of image files based on the download popularity value of image files in the resource pool, thereby dynamically allocating disk I / O and bandwidth resources according to demand, improving the overall performance of image file downloads, and optimizing the user experience of the cloud platform.

[0055] The present application will now be described in detail with reference to the accompanying drawings and embodiments.

[0056] Figure 1 This is a flowchart of a mirrored storage scheduling method based on resource popularity proposed in an embodiment of this application. The method is applied to a public cloud platform, where the resource pool of the public cloud platform includes multiple computing nodes; such as... Figure 1 As shown, the method includes:

[0057] S1: After the target image file is created, a corresponding number of computing nodes are selected from the resource pool based on a first preset value to serve as storage nodes for the target image file; a copy of the target image file is stored in each storage node; wherein the target image file is stored in each storage node in the form of image shards;

[0058] S2: During the download process of the target image file, the download popularity value of the target image file is obtained according to a preset time period; based on the download popularity value, the number of copies of the target image file stored in the resource pool is expanded or reduced.

[0059] In a cloud platform, compute nodes are servers used by users to create cloud hosts. They primarily allocate resources such as CPU (Central Processing Unit) and memory needed by the user's virtual machine. During the selection of the host machine (i.e., compute node), each compute node is equipped with a data disk of a certain size. This disk space is mainly used to store the host machine's (i.e., compute node's) own system, host machine agent software logs, etc. Since users typically use cloud hosts in the form of cloud disks, the disk space utilization of the data disks on compute nodes is low, with a large amount of unused disk space. However, a cloud platform's resource pool often contains hundreds or even thousands of compute node servers. Therefore, in this embodiment, the remaining space on the compute node's data disk is used to store user image files, improving the disk space utilization of the compute node. This eliminates the need to deploy additional high-capacity storage servers to store image files, saving on cloud platform setup costs.

[0060] In this embodiment, after the target image file is created, it is stored in a resource pool according to a preset number of storage copies. Specifically, based on a first preset value, a corresponding number of computing nodes are selected from the resource pool as storage nodes, and one copy of the target image file is stored in each storage node. When storing the target image file, it is divided into image fragments for storage. Thus, when downloading the target image file, the user can download different parts of the target image file from different computing nodes, achieving parallel download and improving the download speed of the target image file.

[0061] Furthermore, during the download of the target image file, the download popularity value of the target image file is obtained at preset time intervals. Based on the download popularity value of the target image file in the previous time period, the popularity of the target image file is judged, and then the appropriate number of storage copies in the resource pool for the current time period is determined based on the download popularity value of the previous time period. The number of target image files stored in the resource pool is then increased or decreased. By dynamically scheduling the number of storage copies of the target image file in the resource pool according to the download popularity value, the number of storage copies of the target image file in the resource pool is increased for target image files with high download popularity values, and the number of storage copies of the target image file in the resource pool is decreased for target image files with low download popularity values. In this way, while providing high-speed concurrent image file downloads, the redundancy of image files stored in the resource pool is reasonably adjusted to avoid storing target image files that exceed the user's download demand, thus avoiding the waste of disk space on computing nodes.

[0062] As one embodiment of this application, selecting a corresponding number of computing nodes from the resource pool based on a first preset value to serve as storage nodes for the target image file includes:

[0063] Obtain the current remaining disk capacity of each computing node in the resource pool, and sort the computing nodes from largest to smallest according to the remaining disk capacity to obtain the sorting result;

[0064] Based on the first preset value, select the top-ranked computing nodes in the sorting results as the storage nodes for the target image file.

[0065] In one embodiment, the compute nodes with the largest remaining disk space in the resource pool are selected as the storage nodes for the target image file. Specifically, the remaining disk capacity of each compute node in the resource pool is obtained, and the compute nodes are sorted from largest to smallest according to their remaining disk capacity to obtain a sorting result. Starting from the first compute node in the sorting result, a corresponding number of compute nodes are selected as the storage nodes for the target image file based on a first preset value. In this embodiment, the first preset value can be set according to the needs of the actual application. In this embodiment, the first preset value is set to 3, that is, when a new target image file is created, it is initially stored in three compute nodes in the resource pool.

[0066] In this embodiment, after each creation of a new target image file, multiple computing nodes with larger remaining disk space in the resource pool are selected as storage nodes for the target image file. This allows for the even consumption of the remaining free space on the data disks of each computing node in the resource pool, preventing some computing nodes from consuming their remaining data disk space too quickly and causing an uneven load on each computing node.

[0067] In one embodiment of this application, each computing node includes multiple data disks; storing one copy of the target image file in each storage node includes:

[0068] The target image file is split into multiple image fragments according to a preset fragment size;

[0069] For each storage node, all image fragments of the target image file are distributed and stored on various data disks of the storage node.

[0070] Currently, public clouds offer the function of creating full-machine images of cloud servers, supporting only raw image format. In this case, a single image file is very large. Compute nodes within the resource pool are interconnected via 10 Gigabit switches, with a maximum throughput of 10,000 Mb / s for a single network interface card (NIC). The data disks used to store the images are SATA disks (Serial Advanced Technology Attachment hard disks), with a single disk I / O throughput of 1,000 Mb / s. For cloud server image download requests, the maximum transmission speed is limited by the bandwidth of the compute node's NIC, resulting in a maximum transfer speed of 10,000 Mb / s. That is, the download speed of the target image file is limited by multiple factors, including NIC throughput, compute node bandwidth, and single disk I / O throughput. Therefore, this embodiment divides the target image file into image slices and distributes these slices across multiple data disks on the compute nodes. This fully utilizes the I / O throughput performance of the NICs, compute nodes, and data disks to maximize the amount of data downloaded in parallel, thereby improving the overall efficiency of image download.

[0071] Therefore, to improve the image download speed, in one embodiment, the target image file is fragmented and stored on various data disks of the storage node. When fragmenting, the size of a single image fragment and the number of fragments in the target image file need to be considered. Smaller individual image fragments result in a larger number of fragments, leading to faster parallel download speeds for the image file. However, too many fragments increase file management complexity; therefore, a suitable image fragment size needs to be set. In this embodiment, the preset fragment size is set to 100MB to 200MB.

[0072] Since a compute node comprises multiple data disks, and each data disk has an upper limit on its IO (Input / Output) throughput, to fully utilize the IO throughput performance of each data disk, this embodiment splits the target image file according to a preset fragment size and distributes the resulting image fragments across multiple data disks on the compute node. By using distributed storage, the IOPS (Input / Output Operations Per Second) of the compute node's disks is fully utilized, enabling parallel downloading of image fragments within a single compute node. Compared to storing all image fragments on a single data disk, which is limited by the IO throughput of that single data disk, distributed storage reduces the access load on a single data disk, further improving the download efficiency of the target image file.

[0073] As one embodiment of this application, obtaining the download popularity value of the target image file according to a preset time period includes:

[0074] According to a preset time period, obtain information about all storage nodes that store the target image file;

[0075] Based on the information of all storage nodes, the image download logs of each storage node are obtained and parsed to obtain the download volume of the target image file within the time period, and the download volume is used as the download popularity value of the target image file.

[0076] In one embodiment, the image popularity data is derived from the analysis of download logs of compute nodes. The download popularity value of the target image file in the previous time period is obtained by parsing the download logs of each compute node. Specifically, firstly, information on all compute nodes (storage nodes) storing the target image file in the current resource pool is determined. Then, the image download logs of each storage node are obtained and parsed to obtain the number of downloads of the target image file in the previous time period. A summation operation is performed based on the download counts of each compute node to obtain the total download volume of the target image file in the resource pool in the previous time period. This total download volume is used as the download popularity value of the target image file in the previous time period.

[0077] As one embodiment of this application, based on the download popularity value, the number of copies of the target image file stored in the resource pool is expanded or reduced, including:

[0078] Obtain the download popularity value within the previous time period, and compare the download popularity value with a popularity threshold; if the download popularity value is greater than the popularity threshold, determine that the target image file is highly popular; if the download popularity value is less than or equal to the popularity threshold, determine that the target image file is low popular.

[0079] When the target image file is highly popular, a target number of copies not less than a second preset value is set as the expected number of copies of the target image file to be stored within the current time period; the second preset value is determined based on the maximum network throughput rate of the resource pool, the maximum transmission rate of the computing node, and the maximum read / write rate of a single data disk; the second preset value is greater than the first preset value.

[0080] When the target image file is of low popularity, the target number of copies of the first preset value is set as the expected number of copies of the target image file to be stored in the current time period.

[0081] Based on the expected number of copies of the target image file to be stored and the actual number of copies stored within the current time period, the number of copies of the target image file stored in the resource pool is increased or decreased.

[0082] The number of copies of a target image file stored in the resource pool directly affects the overall download speed of that target image file. The more copies stored, the more computing nodes can download in parallel, and the higher the overall download efficiency. However, the more copies stored, the more storage space is occupied. Therefore, it is necessary to manage and control the number of copies of each image file in a reasonable manner.

[0083] Figure 2 This is a flowchart illustrating the scheduling of the number of copies of the target image file in one embodiment of this application. For example... Figure 2 As shown, in one embodiment, after creating the target image file, the image file is split and stored in fragments, and the image is initialized to provide a download service. During the user download process, the download popularity value of the image file is obtained according to a time period. The download popularity value of the previous time period is compared with a popularity threshold. If the download popularity value of the previous time period is greater than the popularity threshold, it is determined to be a high-popularity image; if the download popularity value of the previous time period is less than or equal to the popularity threshold, it is determined to be a low-popularity image. For high-popularity images, as many copies as possible need to be stored to improve the speed of concurrent downloads by users. For low-popularity images, since the download volume is smaller, it is not necessary to store too many copies to occupy storage space and affect the download efficiency of other high-popularity images. In practical applications, the popularity threshold can be set according to needs. In this embodiment, the popularity threshold is set to 100.

[0084] For low-popularity image files, a first preset value is used as the number of storage copies for each low-popularity image file, maintaining the initial number of storage copies (e.g., 3 copies) after the target image file is created. For high-popularity image files, a second preset value limits the minimum number of storage copies to fully utilize the throughput performance of the network interface card (NIC), compute nodes, and individual data disks. Specifically, the second preset value is determined based on the cloud platform's network bandwidth, compute node bandwidth, and the maximum read / write rate of a single data disk. For example, consider a NIC maximum throughput of 10000Mb / s, a compute node maximum transmission speed of 10000Mb / s, and a data disk single-disk IO throughput of 1000Mb / s. Disk IO throughput and network (compute node) transmission speed are two bottlenecks; whichever bottleneck is reached first will limit the upper limit of download speed. Due to the transmission speed limitations of the NIC and compute nodes, theoretically, the maximum parallel download speed (i.e., 10000Mb / s) can be provided when the number of data disks used for parallel download reaches 10000 / 1000 = 10. That is, setting the second preset value to 10 ensures that the maximum download speed is achieved when the number of copies of the target image file stored in the resource pool reaches 10 (i.e., stored on 10 different computing nodes). Therefore, in this embodiment, for highly popular target image files, a target number of copies not less than the second preset value is set as the expected number of copies to be stored. In other words, when the number of copies stored is greater than or equal to the second preset value, the total download speed can theoretically reach the maximum network throughput. If the number of copies stored is less than the second preset value, the total download speed is only affected by the single disk I / O throughput performance.

[0085] In this embodiment, given the limitation of IO throughput performance of a single data disk, the computing nodes associated with the image file are expanded to calculate a second preset value. Based on the second preset value, the number of storage copies of the high-popularity image is set to make full use of the throughput performance of the network card, computing nodes and a single data disk, thereby improving the overall download efficiency.

[0086] As one embodiment of this application, when the download popularity value is greater than the popularity threshold, it further includes:

[0087] Compare the target number of portions with a third preset value;

[0088] If the target number of copies is greater than the third preset value, the third preset value will be used as the expected number of copies of the target image file to be stored within the current time period.

[0089] In one embodiment, storing too many copies of a single image file in the resource pool can affect the dynamic scheduling and download speed of other image files. Therefore, it is necessary to control the upper limit of the number of image files stored in the resource pool. In this embodiment, a third preset value is set to limit the upper limit of the number of stored copies. If the target number of copies of the target image file exceeds the third preset value, the third preset value is used as the expected number of copies of the target image file. For example, if the third preset value is set to 15, and the target number of copies is 16, the target image file can be stored in the resource pool at a maximum of 15 copies.

[0090] As one implementation of this application, based on the expected number of storage copies and the actual number of storage copies of the target image file within the current time period, the number of copies of the target image file stored in the resource pool is expanded or reduced, including:

[0091] Calculate the difference between the expected number of copies of the target image file and the actual number of copies stored within the current time period;

[0092] If the difference is positive, the difference is taken as the number of copies to be expanded; based on the number of copies to be expanded, a corresponding number of computing nodes are selected from the resource pool as new storage nodes, and the target image file is stored in the new storage nodes;

[0093] If the difference is negative, the absolute value of the difference is taken as the number of copies to be reduced; based on the number of copies to be reduced, a corresponding number of storage nodes are selected and the target image file in the storage nodes is deleted.

[0094] In one embodiment, the difference between the expected number of storage copies and the actual number of target image files currently stored in the resource pool is calculated, thus determining the number of copies that need to be expanded or reduced. Specifically, if the expected number of storage copies is greater than the actual number, the difference is used as the number of copies to be expanded. Based on this difference, a corresponding number of compute nodes are selected from the resource pool as new storage nodes, and the target image files are stored in each of these new storage nodes. If the expected number of storage copies is less than the actual number, the absolute value of the difference is used as the number of copies to be reduced. Based on this difference, a corresponding number of storage nodes are selected from the storage nodes currently storing the target image files, and the target image files stored on them are deleted, completing the reduction of the number of storage copies.

[0095] Optionally, when selecting the storage node where the target image file needs to be deleted, the download volume of all image files in the previous time period of each storage node is first obtained, and the storage node with the larger download volume of image files in the previous time period is selected to delete the target image file stored on it. In this way, while reducing the number of target image files, the access load of computing nodes with large access volume is reduced, and the load balance of each computing node in the resource pool is improved.

[0096] In one embodiment, some computing nodes store multiple high-popularity image files, leading to excessive node access load and reduced image download efficiency. Conversely, some computing nodes store image files with low popularity, resulting in wasted network bandwidth due to idle time. Therefore, according to a preset time period, the download popularity value of the image files stored on each computing node in the resource pool is obtained, and it is determined whether the image file is high-popularity. The number of high-popularity image files and the number of low-popularity image files in each computing node are counted. Further, the computing node with the most high-popularity image files is designated as the node to be transferred out, and the computing node with the fewest high-popularity image files is designated as the node to be transferred in. A connection is established between the nodes to be transferred out and the nodes to be transferred in. The node to be transferred out transfers at least one high-popularity image file from its local storage to the node to be transferred in, and the node to be transferred in transfers the same number of low-popularity image files from its local storage to the node to be transferred out.

[0097] By transferring high-traffic image files between compute nodes with high access load and low-traffic compute nodes with low access load, the access load of the two nodes is balanced, further improving the overall image download efficiency of all compute nodes in the resource pool.

[0098] As one embodiment of this application, storing the target image file into the new storage node includes:

[0099] Establish point-to-point communication links between all storage nodes currently storing the target image file and all new storage nodes;

[0100] Based on the point-to-point communication link, the target image file is transmitted to all new storage nodes through all storage nodes currently storing the target image file, so that each new storage node can receive different image fragments of the target image file in parallel.

[0101] In one embodiment, the target image file is synchronized among various computing nodes via point-to-point communication. Specifically, information about all storage nodes currently storing the target image file is obtained through a P2P server. Based on this information, point-to-point communication links are established between the old and new storage nodes. All old storage nodes send image fragments of the target image file stored locally to the new storage nodes, enabling each new storage node to acquire different parts of the target image file in parallel, thereby improving the efficiency of dynamic scheduling of image files in the resource pool.

[0102] As one embodiment of this application, the process of providing the target image file for download further includes:

[0103] When any computing node receives a download request for the target image file sent by the client, it searches for information on all computing nodes in the resource pool that store the target image file based on the information of the target image file.

[0104] Based on the information of each computing node that stores the target image file, a point-to-point communication link is established between the client and each computing node, so that the client can download different image fragments of the target image file in parallel from each computing node.

[0105] In one embodiment, a peer-to-peer (P2P) approach is used for multi-threaded downloading to improve download efficiency. P2P download technology fully leverages the advantages of image file fragmentation, distributed storage, and redundant images, maximizing the throughput capacity of each computing node and data disk to achieve high-speed transmission. P2P technology is a communication protocol for sharing files on peer-to-peer networks. When downloading an image file via peer-to-peer, the client first parses the torrent file to obtain the tracking address, then connects to the peer-to-peer server. The peer-to-peer server finds and provides the addresses of all storage nodes for the image file, establishing peer-to-peer communication with the client to achieve parallel downloading.

[0106] Because of the decentralized nature of peer-to-peer communication, downloaders on a local area network can not only receive file fragments but also provide their own file fragments to other downloaders. This allows for more threads to download files in parallel. Therefore, in the case of multiple users downloading concurrently, the clients also connect to each other and exchange data blocks that the others do not have, further improving the overall download efficiency.

[0107] Compared to existing methods that use separate storage servers to manage images, this application saves disk space required for image storage, reducing cloud platform construction costs. Simultaneously, by dynamically scheduling the number of image files stored to adapt to varying user download demands at different times, it improves the overall download performance of the cloud platform. Furthermore, by introducing a P2P download method, it fully leverages the throughput performance of each computing node and data disk, providing faster download speeds and achieving the goal of cost reduction and speed improvement.

[0108] Based on the same inventive concept, one embodiment of this application provides a mirrored storage scheduling system based on resource popularity. (Reference) Figure 3 , Figure 3 This is a schematic diagram of a mirrored storage scheduling system based on resource popularity proposed in an embodiment of this application. Figure 3 As shown, the system includes: a cloud platform base, a peer-to-peer server, and a management platform;

[0109] The cloud platform base includes multiple computing nodes as provided in the above embodiments, and each computing node has a data disk with disk space for storing target image files.

[0110] The peer-to-peer server is used to collect statistics on the information of the image files stored in each computing node; and to provide information on the corresponding computing nodes for establishing peer-to-peer communication links in the resource pool.

[0111] The management platform is used to execute the steps in the method provided in the above embodiments, manage the computing nodes and image files in the resource pool, specify the storage nodes for each newly created image file, and dynamically schedule the number of copies of each image file stored in the resource pool according to the download popularity value of each image file.

[0112] In this embodiment, the system includes a cloud platform base, peer-to-peer (P2P) servers, and a management platform. The cloud platform base comprises a resource pool consisting of multiple computing nodes that communicate with each other via peer-to-peer communication. The cloud platform base does not have a dedicated storage server; instead, storage space is allocated from the data disks of each computing node to store image fragments of user image files and provide download services. Users create private images on the cloud platform through uploading and cloud host creation. The management platform then divides the image files into multiple image fragments and stores them on the data disks of the multiple computing nodes.

[0113] The management platform is used to manage image instances in the resource pool (resource nodes), execute image sharding algorithms, and dynamically schedule image storage.

[0114] In the system, point-to-point proxy services are deployed on each compute node to communicate with the point-to-point server. The point-to-point server continuously collects information about locally stored image files uploaded by each compute node, as well as the status information of each compute node. Compute nodes in the resource pool communicate with each other via point-to-point. When a client requests to download an image file from a compute node, it obtains information about all compute nodes storing that image file from the point-to-point server, establishes point-to-point communication links between the client and each compute node, and then simultaneously downloads different parts of the image file from each compute node. The management platform dynamically adjusts the number of image copies stored in the resource pool and the distribution of each image file across compute nodes based on the download popularity of the image files, fully utilizing the I / O performance of each disk to improve concurrent download speeds and enhance overall system performance.

[0115] Furthermore, given the file-sharing feature of peer-to-peer communication, when multiple clients are downloading image files concurrently, each client can also share the image fragments it has downloaded, further improving the efficiency of parallel image file downloads.

[0116] As one embodiment of this application, the management platform is used to select a corresponding number of computing nodes from the resource pool based on a first preset value after the target image file is created, as storage nodes for the target image file; and control each storage node to store a copy of the target image file; wherein the target image file is stored in each storage node in the form of image fragments;

[0117] The peer-to-peer server is used to obtain information about the image files uploaded by each computing node according to a preset time period; the information about the image files includes: the storage location of each image fragment, and the number of times each image file was downloaded in the previous time period.

[0118] The management platform is also used to obtain the number of downloads of the target image file from the peer-to-peer server during the download process of the target image file, determine the download popularity value of the target image file based on the number of downloads, and expand or reduce the number of storage copies of the target image file stored in the resource pool based on the download popularity value.

[0119] As one embodiment of this application, the management platform is configured to select a corresponding number of computing nodes from the resource pool based on a first preset value, as storage nodes for the target image file, specifically including:

[0120] Obtain the current remaining disk capacity of each computing node in the resource pool, and sort the computing nodes from largest to smallest according to the remaining disk capacity to obtain the sorting result;

[0121] Based on the first preset value, select the top-ranked computing nodes in the sorting results as the storage nodes for the target image file.

[0122] In one embodiment of this application, each computing node includes multiple data disks; the management platform is used to control each storage node to store a copy of the target image file, specifically including:

[0123] The target image file is split into multiple image fragments according to a preset fragment size;

[0124] For each storage node, all image fragments of the target image file are distributed and stored on various data disks of the storage node.

[0125] As one embodiment of this application, the peer-to-peer server is specifically used to obtain information about all storage nodes storing the target image file according to a preset time period.

[0126] The management platform is used to obtain and parse the image download logs of each storage node based on the information of all storage nodes, obtain the download volume of the target image file within the time period, and use the download volume as the download popularity value of the target image file.

[0127] As one embodiment of this application, the management platform is used to expand or reduce the number of target image files stored in the resource pool based on the download popularity value, specifically including:

[0128] Obtain the download popularity value within the previous time period, and compare the download popularity value with a popularity threshold; if the download popularity value is greater than the popularity threshold, determine that the target image file is highly popular; if the download popularity value is less than or equal to the popularity threshold, determine that the target image file is low popular.

[0129] When the target image file is highly popular, a target number of copies not less than a second preset value is set as the expected number of copies of the target image file to be stored within the current time period; the second preset value is determined based on the maximum network throughput rate of the resource pool, the maximum transmission rate of the computing node, and the maximum read / write rate of a single data disk; the second preset value is greater than the first preset value.

[0130] When the target image file is of low popularity, the target number of copies of the first preset value is set as the expected number of copies of the target image file to be stored in the current time period.

[0131] Based on the expected number of copies of the target image file to be stored and the actual number of copies stored within the current time period, the number of copies of the target image file stored in the resource pool is increased or decreased.

[0132] As one embodiment of this application, when the download popularity value is greater than the popularity threshold, the management platform is further configured to perform the following steps:

[0133] Compare the target number of portions with a third preset value;

[0134] If the target number of copies is greater than the third preset value, the third preset value will be used as the expected number of copies of the target image file to be stored within the current time period.

[0135] As one embodiment of this application, the management platform is used to expand or reduce the number of target image files stored in the resource pool based on the expected number of storage copies and the actual number of storage copies of the target image files within the current time period, specifically including:

[0136] Calculate the difference between the expected number of copies of the target image file and the actual number of copies stored within the current time period;

[0137] If the difference is positive, the difference is taken as the number of copies to be expanded; based on the number of copies to be expanded, a corresponding number of computing nodes are selected from the resource pool as new storage nodes, and the target image file is stored in the new storage nodes;

[0138] If the difference is negative, the absolute value of the difference is taken as the number of copies to be reduced; based on the number of copies to be reduced, a corresponding number of storage nodes are selected and the target image file in the storage nodes is deleted.

[0139] As one embodiment of this application, the management platform is used to store the target image file to the new storage node, specifically including:

[0140] The system controls the peer-to-peer server to obtain information about all storage nodes that currently store the target image file; and establishes peer-to-peer communication links between all storage nodes that currently store the target image file and all new storage nodes.

[0141] Based on the point-to-point communication link, control all storage nodes currently storing the target image file to transmit the target image file to all new storage nodes, so that each new storage node can receive different image fragments of the target image file in parallel.

[0142] In one embodiment of this application, the peer-to-peer server is further configured to perform the following steps:

[0143] When any computing node receives a download request for the target image file sent by the client, it searches for information on all computing nodes in the resource pool that store the target image file based on the information of the target image file.

[0144] Based on the information of each computing node that stores the target image file, a point-to-point communication link is established between the client and each computing node, so that the client can download different image fragments of the target image file in parallel from each computing node.

[0145] Regarding the system in the above embodiments, the specific ways in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0146] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0147] For the sake of simplicity, the method embodiments are described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and components involved are not necessarily essential to this application.

[0148] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0149] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0150] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0151] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0152] Although preferred embodiments of the embodiments of this application have been described, those skilled in the art, once they understand the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, this application is to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of this application.

[0153] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0154] The above provides a detailed description of the resource-based mirrored storage scheduling method and system provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A resource popularity-based image storage scheduling method, characterized in that, The method is applied to a public cloud platform, wherein the resource pool of the public cloud platform includes multiple computing nodes; the method includes: After the target image file is created, a corresponding number of computing nodes are selected from the resource pool based on a first preset value as the storage nodes of the target image file. Specifically, this includes: obtaining the current remaining disk capacity of each computing node in the resource pool, and sorting each computing node according to the remaining disk capacity from largest to smallest to obtain a sorting result; and selecting the top computing nodes in the sorting result as the storage nodes of the target image file based on the first preset value. Each storage node stores one copy of the target image file; wherein the target image file is stored in each storage node in the form of image fragments; During the download process of the target image file, the download popularity value of the target image file is obtained according to a preset time period. Specifically, this includes: obtaining information of all storage nodes storing the target image file according to the preset time period; obtaining and parsing the image download log of each storage node based on the information of all storage nodes to obtain the download volume of the target image file within the time period, and using the download volume as the download popularity value of the target image file. Based on the download popularity value, the number of copies of the target image file stored in the resource pool is increased or decreased.

2. The mirrored storage scheduling method based on resource popularity according to claim 1, characterized in that, Each compute node includes multiple data disks; The step of storing one copy of the target image file in each storage node includes: The target image file is split into multiple image fragments according to a preset fragment size; For each storage node, all image fragments of the target image file are distributed and stored on various data disks of the storage node.

3. The mirrored storage scheduling method based on resource popularity according to claim 1, characterized in that, Based on the download popularity value, the number of copies of the target image file stored in the resource pool is increased or decreased, including: Obtain the download popularity value within the previous time period, and compare the download popularity value with a popularity threshold; if the download popularity value is greater than the popularity threshold, determine that the target image file is highly popular; if the download popularity value is less than or equal to the popularity threshold, determine that the target image file is low popular. When the target image file is highly popular, a target number of copies not less than a second preset value is set as the expected number of copies of the target image file to be stored within the current time period; the second preset value is determined based on the maximum network throughput rate of the resource pool, the maximum transmission rate of the computing node, and the maximum read / write rate of a single data disk; the second preset value is greater than the first preset value. When the target image file is of low popularity, the target number of copies of the first preset value is set as the expected number of copies of the target image file to be stored in the current time period. Based on the expected number of copies of the target image file to be stored and the actual number of copies stored within the current time period, the number of copies of the target image file stored in the resource pool is increased or decreased.

4. The mirrored storage scheduling method based on resource popularity according to claim 3, characterized in that, If the download popularity value is greater than the popularity threshold, the method further includes: Compare the target number of portions with a third preset value; If the target number of copies is greater than the third preset value, the third preset value will be used as the expected number of copies of the target image file to be stored within the current time period.

5. The mirrored storage scheduling method based on resource popularity according to claim 1, characterized in that, Based on the expected and actual number of copies of the target image file stored within the current time period, the number of copies of the target image file stored in the resource pool is expanded or reduced, including: Calculate the difference between the expected number of copies of the target image file and the actual number of copies stored within the current time period; If the difference is positive, the difference is taken as the number of copies to be expanded; based on the number of copies to be expanded, a corresponding number of computing nodes are selected from the resource pool as new storage nodes, and the target image file is stored in the new storage nodes; If the difference is negative, the absolute value of the difference is taken as the number of copies to be reduced; based on the number of copies to be reduced, a corresponding number of storage nodes are selected and the target image file in the storage nodes is deleted.

6. The mirrored storage scheduling method based on resource popularity according to claim 5, characterized in that, Storing the target image file into the new storage node includes: Establish point-to-point communication links between all storage nodes currently storing the target image file and all new storage nodes; Based on the point-to-point communication link, the target image file is transmitted to all new storage nodes through all storage nodes currently storing the target image file, so that each new storage node can receive different image fragments of the target image file in parallel.

7. The mirrored storage scheduling method based on resource popularity according to claim 1, characterized in that, The process of providing the download of the target image file also includes: When any computing node receives a download request for the target image file sent by the client, it searches for information on all computing nodes in the resource pool that store the target image file based on the information of the target image file. Based on the information of each computing node that stores the target image file, a point-to-point communication link is established between the client and each computing node, so that the client can download different image fragments of the target image file in parallel from each computing node.

8. A mirrored storage scheduling system based on resource popularity, characterized in that, include: Cloud platform infrastructure, peer-to-peer servers, and management platform; The cloud platform base includes multiple computing nodes as described in any one of claims 1-7, and each computing node has a data disk with disk space for storing the target image file. The peer-to-peer server is used to collect statistics on the information of the image files stored in each computing node; and to provide information on the corresponding computing nodes for establishing peer-to-peer communication links in the resource pool. The management platform is used to perform the steps in the method as described in any one of claims 1-7, to manage the computing nodes and image files in the resource pool, to specify the storage nodes for each newly created image file, and to dynamically schedule the number of copies of each image file stored in the resource pool based on the download popularity value of each image file.

Images