Mirror image storage scheduling method and system based on resource popularity

Abstract

The invention relates to the field of cloud computing, and discloses a mirror image storage scheduling method and system based on resource popularity. The method is applied to a public cloud platform, and a resource pool of the public cloud platform comprises a plurality of computing nodes. The method comprises the following steps: after a target mirror image file is created, selecting a corresponding number of computing nodes from a resource pool based on a first preset value as storage nodes of the target mirror image file; one target mirror image file is stored in each storage node; wherein the target mirror image file is stored in each storage node in a mirror image fragment form; in the process of providing downloading of the target mirror image file, obtaining a downloading popularity value of the target mirror image file according to a preset time period; and based on the downloading popularity value, expanding or reducing the number of the target mirror image files stored in the resource pool. By adopting the method, the efficiency of downloading the mirror image file of the cloud platform can be improved, and the construction cost of the cloud platform is saved.

Description

Technical Field

[0001] The present application relates to the field of cloud computing, and in particular to a method and system for scheduling mirror storage based on resource heat. Background Art

[0002] In the public cloud platform, the image service is a basic product function of the cloud platform. The cloud platform service provides the ability to manage the lifecycle of images. Users can flexibly use public images, private images, or shared images to apply for elastic cloud hosts and physical machines. In addition, users can also create private images through existing cloud hosts or use external image files to achieve business cloud or cloud migration. Through the image service, users can upload private images to object storage and download them locally, or specify images to create cloud hosts and import files from object storage to create images. In addition, users can share images and provide them to other users for download and use. During the transmission, download, and storage of image files, the upload and download speeds directly affect the user experience of the cloud platform users.

[0003] The number of users on public cloud platforms is huge. In the resource pool of the cloud platform, each user has a certain amount of private image quota, that is, the user can create a private image and store it in the resource pool. In order to meet the needs of users for image storage and downloading, the space required to store image files in the resource pool is very large. Usually, a dedicated large-capacity storage server is deployed to store user image files, and the cost of building a cloud platform is relatively high. In order to ensure user experience, while providing large-capacity storage, the rate at which users download image files must also be considered. However, in the scenario where multiple users download image files concurrently, due to the bandwidth limitation of the storage server, the user's image download rate is low, which seriously affects the user experience.

[0004] Based on this, how to improve the efficiency of concurrent downloading of image files by multiple users, cope with high concurrent download requirements, and improve user experience is a problem that needs to be solved at present. Summary of the invention

[0005] In view of this, the present application aims to propose a method and system for image storage scheduling based on resource heat, so as to improve the efficiency of users' concurrent downloading of image files and enhance the user experience of the public cloud platform.

[0006] To achieve the above purpose, the technical solution of this application is as follows:

[0007] In a first aspect, an embodiment of the present application provides a method for scheduling image storage based on resource heat, the method being applied to a public cloud platform, wherein a resource pool of the public cloud platform includes a plurality of computing nodes; the method comprising:

[0008] After the target image file is created, a corresponding number of computing nodes are selected from the resource pool based on a first preset value as storage nodes for the target image file; a copy of the target image file is stored in each storage node; wherein the target image file is stored in each storage node in the form of image slices;

[0009] In the process of providing the target image file for download, the download heat value of the target image file is obtained according to a preset time period; based on the download heat value, the number of copies of the target image file stored in the resource pool is expanded or reduced.

[0010] Optionally, selecting a corresponding number of computing nodes from the resource pool based on a first preset value as storage nodes for the target image file includes:

[0011] Obtaining the current remaining disk capacity of each computing node in the resource pool, and sorting the computing nodes from large to small according to the remaining disk capacity to obtain a sorting result;

[0012] Based on the first preset value, a plurality of computing nodes at the top of the sorting result are selected as storage nodes for the target image file.

[0013] Optionally, each computing node includes multiple data disks; storing a copy of the target image file in each storage node includes:

[0014] According to a preset fragment size, the target image file is split into multiple image fragments;

[0015] For each storage node, all image fragments of the target image file are dispersedly stored in each data disk of the storage node.

[0016] Optionally, obtaining the download heat value of the target image file according to a preset time period includes:

[0017] According to a preset time period, obtain information of all storage nodes storing the target image file;

[0018] Based on the information of all storage nodes, the image download log of each storage node is obtained and parsed to obtain the download amount of the target image file in the time period, and the download amount is used as the download heat value of the target image file.

[0019] Optionally, based on the download heat value, the number of copies of the target image file stored in the resource pool is expanded or reduced, including:

[0020] Obtaining a download heat value in the previous time period, and comparing the download heat value with a heat threshold; if the download heat value is greater than the heat threshold, determining that the target image file is of high heat; if the download heat value is less than or equal to the heat threshold, determining that the target image file is of low heat;

[0021] In the case where the target image file is of high popularity, a target number of copies not less than a second preset value is set as the expected number of stored copies of the target image file in the current time period; the second preset value is determined based on the maximum network throughput rate of the resource pool, the maximum transmission rate of the computing node, and the maximum read and write rate of a single data disk; the second preset value is greater than the first preset value;

[0022] When the target image file has low heat, setting the target number of copies of the first preset value as the expected number of stored copies of the target image file in the current time period;

[0023] Based on the expected number of stored copies and the actual number of stored copies of the target image file in the current time period, the number of copies of the target image file stored in the resource pool is expanded or reduced.

[0024] Optionally, when the download heat value is greater than the heat threshold, the method further includes:

[0025] comparing the target number of copies with a third preset value;

[0026] When the target number of copies is greater than the third preset value, the third preset value is used as the expected number of stored copies of the target image file in the current time period.

[0027] Optionally, based on the expected number of stored copies and the actual number of stored copies of the target image file in the current time period, the number of copies of the target image file stored in the resource pool is expanded or reduced, including:

[0028] Calculate the difference between the expected number of stored copies of the target image file and the actual number of stored copies in the current time period;

[0029] When the difference is a positive number, the difference is used as the number of copies that need to be expanded; based on the number of copies that need to be expanded, a corresponding number of computing nodes are selected from the resource pool as new storage nodes, and the target image file is stored in the new storage nodes;

[0030] When the difference is a negative number, the absolute value of the difference is used as the number of copies that need to be reduced; based on the number of copies that need to be reduced, a corresponding number of storage nodes are selected and the target image files in the storage nodes are deleted.

[0031] Optionally, storing the target image file in the new storage node includes:

[0032] Establishing a point-to-point communication link between all storage nodes currently storing the target image file and all new storage nodes;

[0033] Based on the point-to-point communication link, the target image file is transmitted to all new storage nodes through all storage nodes currently storing the target image file, so that each new storage node receives different image slices of the target image file in parallel.

[0034] Optionally, in the process of providing the target image file for download, the process further includes:

[0035] When any computing node receives a download request for the target image file sent by the client, searching for information of all computing nodes storing the target image file in the resource pool according to information of the target image file;

[0036] Based on the information of each computing node storing the target image file, a point-to-point communication link is established between the client and each computing node, so that the client can download different image fragments of the target image file from each computing node in parallel.

[0037] According to a second aspect of an embodiment of the present application, there is provided a mirror storage scheduling system based on resource heat, the system comprising: a cloud platform base, a point-to-point server and a management platform;

[0038] The cloud platform base includes a plurality of computing nodes as in the method provided in the first aspect of the embodiment of the present application, and a disk space for storing a target image file is provided in a data disk of each computing node;

[0039] The point-to-point server is used to collect statistics on the information of the image files stored in each computing node; and to provide information of the corresponding computing nodes for establishing a point-to-point communication link in the resource pool;

[0040] The management platform is used to execute the steps in the method provided in the first aspect of the embodiment of the present application, manage the computing nodes and image files in the resource pool, specify the storage nodes of each newly created image file, and dynamically schedule the number of copies of each image file stored in the resource pool according to the download heat value of each image file.

[0041] By adopting the image storage scheduling method based on resource heat provided by the present application, after the target image file is created, multiple computing nodes of a first preset value are selected from the resource pool as storage nodes, and the target image file is stored in each storage node in the form of image fragments. In the process of providing the download service of the target image file on the cloud platform, the download heat value of the target image file is obtained according to a certain time period, and the number of copies of the target image file stored in the resource pool is dynamically scheduled based on the download heat value.

[0042] The image storage scheduling method based on resource heat provided by the present application does not use a dedicated large-capacity storage server to store image files, but stores the target image files created by users in the computing nodes of the resource pool. The free storage space in the computing node is used to store the image fragments of the image file, and there is no need to deploy additional storage servers, which saves the cost of building a cloud platform. When storing the target image file, multiple redundant target image files are stored in different computing nodes, so that users can request downloads from different computing nodes. Compared with the method of providing image file downloads through a dedicated large-capacity storage server, this method uses multiple computing nodes to store multiple target image files. When storing the target image file, it is stored in the computing node in the form of image fragments, so that different parts of the image file can be downloaded in parallel, which improves the download speed of the image file, reduces the load pressure of a single computing node, and improves the overall performance of the cloud platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0044] Figure 1 It is a flow chart of a mirror storage scheduling method based on resource heat proposed in an embodiment of the present application;

[0045] Figure 2 This is a flow chart of scheduling the number of copies of a target image file in one embodiment of the present application;

[0046] Figure 3 It is a schematic diagram of a mirror storage scheduling system based on resource heat proposed in an embodiment of the present application. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0048] It should be understood that the references to "one embodiment" or "an embodiment" throughout the specification mean that the specific features, structures, or characteristics associated with the embodiment are included in at least one embodiment of the present application. Therefore, the references to "in one embodiment" or "in an embodiment" appearing throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0049] In the various embodiments of the present application, it should be understood that the size of the serial numbers of the following processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0050] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects as detailed in the present application.

[0051] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application may be combined with each other.

[0052] The image service on the public cloud platform provides official images, private images, and shared images. Among them, the public image is a common standard operating system image that is visible to all users, including the operating system and pre-installed public applications.

[0053] Private images contain images of operating systems or business data, pre-installed public applications, and users' private applications, and are visible only to the user. Private images include system disk images, data disk images, ISO images, and whole machine images. Among them, system disk images contain images of operating systems and application software required for users to run their businesses, and can be used to create cloud hosts and migrate users' businesses to the cloud. Data disk images only contain images of users' business data, and can be used to create cloud hard disks and migrate users' business data to the cloud. ISO images are private images that register the ISO files of external images to the cloud platform. They are special images that can only be issued to cloud hosts used as temporary transitions. Whole machine images are also called full images, and contain images of operating systems, application software, and business data required for users to run their businesses. Whole machine images are made based on differential backups, and are more efficient to create than system disk images and data disk images of the same disk capacity. Users can share the created private images with other users for download and use, which is called shared images.

[0054] The present application provides a method for dynamically scheduling image file storage according to the download heat value of the image file in the resource pool, so as to dynamically allocate disk IO and bandwidth resources according to demand, improve the overall performance of image file downloading, and optimize the user experience of the cloud platform.

[0055] The present application will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0056] Figure 1 : is a flow chart of a method for image storage scheduling based on resource heat proposed in an embodiment of the present application. The method is applied to a public cloud platform, and the resource pool of the public cloud platform includes multiple computing nodes; Figure 1 As shown, the method includes:

[0057] S1: After the target image file is created, a corresponding number of computing nodes are selected from the resource pool based on a first preset value as storage nodes for the target image file; a copy of the target image file is stored in each storage node; wherein the target image file is stored in each storage node in the form of image fragments;

[0058] S2: in the process of providing the target image file for download, obtaining the download heat value of the target image file according to a preset time period; and based on the download heat value, expanding or reducing the number of copies of the target image file stored in the resource pool.

[0059] In the cloud platform, the computing node is a server used by users to apply for the creation of cloud hosts. It is mainly used to allocate the CPU (Central Processing Unit), memory and other resources required by the user's virtual machine. In the selection of the host machine (i.e., computing node), the computing node will be equipped with a data disk of a certain size. The disk space of this part of the data disk is mainly used to store the host machine (i.e., computing node) itself, the host agent software log, etc. Since current users usually use cloud hosts in the form of cloud disks, the disk space utilization rate of the data disks in the computing nodes is low, and a large amount of free disk space is left unused. However, the number of computing node servers in a resource pool of the cloud platform is very large, often reaching hundreds or thousands of machines. Based on this, in this embodiment, the remaining space of the computing node's data disk is used to store the user's image files, which improves the disk space utilization rate of the computing node. At the same time, there is no need to deploy additional large-capacity storage servers to store image files, saving the cost of building the cloud platform.

[0060] In this embodiment, after the target image file is created, the target image file is stored in the resource pool according to the preset number of storage copies. Specifically, based on the first preset value, a corresponding number of computing nodes are selected from the resource pool as storage nodes, and a copy of the target image file is stored in each storage node. When storing the target image file, the target image file is divided into image slices for storage, so that when downloading the target image file, the user can download different parts of the target image file from different computing nodes, realizing parallel downloading and improving the download speed of the target image file.

[0061] In addition, in the process of providing the target image file for user download, the download heat value of the target image file is obtained according to the preset time period, and the heat level of the target image file is judged according to the download heat value of the target image file in the previous time period, and then based on the download heat value of the previous time period, the appropriate number of storage copies in the resource pool in the current time period is determined, and the number of target image files stored in the current resource pool is expanded or reduced. By dynamically scheduling the number of storage copies of the target image file in the resource pool according to the download heat value of the target image file, for the target image file with high download heat value, its storage number in the resource pool is increased, and for the target image file with low download heat value, its storage number in the resource pool is reduced. In this way, while providing high-speed concurrent image file download, the redundancy of the image files stored in the resource pool is reasonably adjusted to avoid storing target image files that exceed the download demand of users, resulting in a waste of disk space of computing nodes.

[0062] As an implementation manner of the present application, selecting a corresponding number of computing nodes from the resource pool based on a first preset value as storage nodes for the target image file includes:

[0063] Obtaining the current remaining disk capacity of each computing node in the resource pool, and sorting the computing nodes from large to small according to the remaining disk capacity to obtain a sorting result;

[0064] Based on the first preset value, a plurality of computing nodes at the top of the sorting result are selected as storage nodes for the target image file.

[0065] In one embodiment, a computing node with a larger current remaining disk space in a resource pool is selected as a storage node for a target image file. Specifically, the current remaining disk capacity of each computing node in the resource pool is obtained, and each computing node is sorted from large to small according to the remaining disk capacity to obtain a sorting result. Starting from the first computing node in the sorting result, a corresponding number of computing nodes are selected as storage nodes for the target image file based on a first preset value. In this embodiment, the first preset value can be set according to the needs of the actual application. In this embodiment, the first preset value is set to 3, that is, when a new target image file is created, it is initially stored in three computing nodes in the resource pool.

[0066] In this embodiment, each time a new target image file is created, multiple computing nodes with larger remaining disk space in the resource pool are selected as storage nodes for the target image file, so that the remaining free space of the data disks of each computing node in the resource pool can be evenly consumed, thereby avoiding the remaining space of the data disks of some computing nodes being consumed too quickly, resulting in unbalanced load on each computing node.

[0067] As an implementation manner of the present application, each computing node includes multiple data disks; storing a copy of the target image file in each storage node includes:

[0068] According to a preset fragment size, the target image file is split into multiple image fragments;

[0069] For each storage node, all image fragments of the target image file are dispersedly stored in each data disk of the storage node.

[0070] Currently, the public cloud provides the function of whole machine mirroring of cloud hosts, and the supported mirroring format is raw format. In this case, a single mirror file is very large. The computing nodes in the resource pool are connected to each other through a 10 Gigabit switch, and the maximum throughput of a single network card is 10000Mb / s. The data disk used for storing the mirror is a SATA disk (Serial Advanced Technology Attachment hard disk), and its single disk IO throughput is 1000Mb / s. For the mirror download request of the cloud host, the maximum transmission speed is 10000Mb / s due to the bandwidth limit of the computing node network card. That is, the download speed of the target mirror file is subject to multiple restrictions on the network card throughput, computing node bandwidth, and data disk single disk IO throughput. Based on this, in this embodiment, the target mirror file is cut into mirror slices, and the mirror slices are dispersed and stored in multiple data disks of the computing node, making full use of the IO throughput performance of the network card, computing node and data disk, and expanding the amount of data downloaded in parallel as much as possible to improve the overall efficiency of mirror download.

[0071] Based on this, in order to improve the speed of image download, in one embodiment, the target image file is fragmented and stored in each data disk of the storage node. When fragmenting, it is necessary to comprehensively consider the size of a single image fragment and the number of fragments of the target image file. The smaller the single image fragment, the more fragments there are, and the faster the parallel download speed of the image file. However, too many fragments will increase the complexity of file management. Therefore, it is necessary to set a moderate image fragment size. In this embodiment, the preset fragment size is set to 100M~200M.

[0072] Since the computing node includes multiple data disks, and each data disk has an upper limit value of IO (Input / Output) throughput, in order to fully utilize the IO throughput performance of each data disk, in this embodiment, the target image file is segmented according to the preset slice size, and the obtained multiple image slices are dispersedly stored in multiple data disks of the computing node. Through the distributed storage method, the IOPS (Input / Output Operations Per Second) of the computing node disk is fully utilized, and the parallel download of the image slices is realized in a single computing node. Compared with the method of storing all image slices in a single data disk subject to the IO throughput limit of a single data disk, the distributed storage reduces the access load of a single data disk, and further improves the download efficiency of the target image file.

[0073] As an implementation manner of the present application, obtaining the download heat value of the target image file according to a preset time period includes:

[0074] According to a preset time period, obtain information of all storage nodes storing the target image file;

[0075] Based on the information of all storage nodes, the image download log of each storage node is obtained and parsed to obtain the download amount of the target image file in the time period, and the download amount is used as the download heat value of the target image file.

[0076] In one embodiment, the heat data of the image is derived from the analysis of the download log of the computing node, and the download heat value of the target image file in the previous time period is obtained by parsing the download log of each computing node. Specifically, first determine the information of all computing nodes (storage nodes) storing the target image file in the current resource pool, and then obtain and parse the image download log of each storage node to obtain the number of downloads of the target image file in the previous time period. Perform a summation operation based on the number of downloads of each computing node to obtain the total download amount of the target image file in the resource pool in the previous time period, and use the total download amount as the download heat value of the target image file in the previous time period.

[0077] As an implementation manner of the present application, based on the download heat value, the number of copies of the target image file stored in the resource pool is expanded or reduced, including:

[0078] Obtaining a download heat value in the previous time period, and comparing the download heat value with a heat threshold; if the download heat value is greater than the heat threshold, determining that the target image file is of high heat; if the download heat value is less than or equal to the heat threshold, determining that the target image file is of low heat;

[0079] In the case where the target image file is of high popularity, a target number of copies not less than a second preset value is set as the expected number of stored copies of the target image file in the current time period; the second preset value is determined based on the maximum network throughput rate of the resource pool, the maximum transmission rate of the computing node, and the maximum read and write rate of a single data disk; the second preset value is greater than the first preset value;

[0080] When the target image file has low heat, setting the target number of copies of the first preset value as the expected number of stored copies of the target image file in the current time period;

[0081] Based on the expected number of stored copies and the actual number of stored copies of the target image file in the current time period, the number of copies of the target image file stored in the resource pool is expanded or reduced.

[0082] The number of copies of the target image file stored in the resource pool directly affects the overall download rate of the target image file. The more copies are stored, the more computing nodes can download in parallel, and the higher the overall download efficiency. However, the more copies are stored, the more storage space is occupied. Therefore, it is necessary to reasonably manage and control the number of copies stored for each image file.

[0083] Figure 2 FIG. 1 is a flow chart for scheduling the number of copies of a target image file in one embodiment of the present application. Figure 2 As shown, in one embodiment, after creating the target image file, the image file is segmented, stored in the form of segments, and the image is initialized to provide download services. In the process of providing users with downloads, the download heat value of the image file is obtained according to the time period, and the download heat value in the previous time period is compared with the heat threshold. When the download heat value in the previous time period is greater than the heat threshold, it is determined to be a high-heat image. When the download heat value in the previous time period is less than or equal to the heat threshold, it is determined to be a low-heat image. For high-heat images, it is necessary to store as many copies as possible to increase the speed of concurrent downloads by users, while for low-heat images, due to the small amount of downloads, it is not necessary to store too many copies to occupy storage space, which affects the download efficiency of other high-heat images. In practical applications, the heat threshold can be set according to demand. In this embodiment, the heat threshold is set to 100.

[0084] For low-heat image files, the first preset value is used as the number of storage copies of each low-heat image file, that is, the initial number of storage copies (for example, 3 copies) of the target image file after it is created is maintained. For high-heat image files, the minimum number of storage copies is limited by the second preset value to fully utilize the throughput performance of the network card, computing node and single data disk. Specifically, the second preset value is determined based on the network bandwidth of the cloud platform, the bandwidth of the computing node and the maximum read and write rate of a single data disk. Take the maximum throughput of the network card of 10000Mb / s, the maximum transmission speed of the computing node of 10000Mb / s, and the single disk IO throughput of the data disk of 1000Mb / s as an example. Disk IO throughput performance and network (computing node) transmission speed are two bottlenecks. No matter which bottleneck is reached first, the upper limit of the download speed will be limited. Due to the transmission speed limit of the network card and the computing node, in theory, when the number of data disks downloaded in parallel reaches 10000 / 1000=10, the maximum parallel download speed (i.e., 10000Mb / s) can be provided. That is, the second preset value is set to 10, and the maximum download speed can be achieved when the number of copies of the target image file stored in the resource pool reaches 10 (that is, stored in 10 different computing nodes). Therefore, in this embodiment, for a target image file with high popularity, a target number of copies not less than the second preset value is set as the expected number of stored copies. That is, when the number of stored copies is greater than or equal to the second preset value, the total download speed can theoretically reach the maximum network throughput. If the number of stored copies is less than the second preset value, the total download speed is only affected by the IO throughput performance of a single disk.

[0085] In this embodiment, in view of the limitation of the IO throughput performance of a single data disk, the computing nodes associated with the image file are expanded to calculate the second preset value, and the number of storage copies of the high-temperature image is set based on the second preset value to fully utilize the throughput performance of the network card, computing nodes and a single data disk, thereby improving the overall download efficiency.

[0086] As an implementation manner of the present application, when the download heat value is greater than the heat threshold, it also includes:

[0087] comparing the target number of copies with a third preset value;

[0088] When the target number of copies is greater than the third preset value, the third preset value is used as the expected number of stored copies of the target image file in the current time period.

[0089] In one embodiment, if too many copies of a certain image file are stored in the resource pool, it will affect the dynamic scheduling and download speed of other image files. Therefore, it is necessary to control the upper limit of the number of copies of the image file stored in the resource pool. In this embodiment, a third preset value is set to limit the upper limit of the number of stored copies. In the case where the target number of copies set for the expanded target image file exceeds the third preset value, the third preset value is used as the expected number of stored copies of the target image file. For example, the third preset value is set to 15. When the target number of copies is 16, the maximum number of copies that can be stored in the resource pool for the target image file is 15.

[0090] As an implementation manner of the present application, based on the expected number of stored copies and the actual number of stored copies of the target image file in the current time period, the number of copies of the target image file stored in the resource pool is expanded or reduced, including:

[0091] Calculate the difference between the expected number of stored copies of the target image file and the actual number of stored copies in the current time period;

[0092] When the difference is a positive number, the difference is used as the number of copies that need to be expanded; based on the number of copies that need to be expanded, a corresponding number of computing nodes are selected from the resource pool as new storage nodes, and the target image file is stored in the new storage nodes;

[0093] When the difference is a negative number, the absolute value of the difference is used as the number of copies that need to be reduced; based on the number of copies that need to be reduced, a corresponding number of storage nodes are selected and the target image files in the storage nodes are deleted.

[0094] In one embodiment, the difference between the expected number of storage copies and the actual number of storage copies of the target image file in the current resource pool is calculated, and the number of copies that need to be expanded or reduced is determined. Wherein, in the case where the expected number of storage copies is greater than the actual number of storage copies, the difference is used as the number of copies that need to be expanded, and based on the difference, a corresponding number of computing nodes are selected from the resource pool as new storage nodes, and the target image file is stored in each new storage node respectively. In the case where the expected number of storage copies is less than the actual number of storage copies, the absolute value of the difference is used as the number of copies that need to be reduced, and based on the difference, a corresponding number of storage nodes are selected from each storage node that currently stores the target image file, and the target image file stored thereon is deleted, thereby completing the operation of reducing the number of storage copies.

[0095] Optionally, when selecting a storage node from which the target image file needs to be deleted, first obtain the download volume of all image files in each storage node in the previous time period, and select a storage node with a large number of image file downloads in the previous time period to delete the target image file stored thereon, thereby reducing the number of copies of the target image file while reducing the access load of the computing node with a large access volume, thereby improving the load balancing of each computing node in the resource pool.

[0096] In one embodiment, some computing nodes store multiple high-heat image files, resulting in excessive node access load and reduced image download efficiency, while the image files stored in some computing nodes are of low heat, and the network bandwidth is idle most of the time, resulting in waste. Based on this, according to a preset time period, the download heat value of the image files stored in each computing node in the resource pool is obtained, and it is determined whether the image file is of high heat. The number of high-heat image files and the number of low-heat image files in each computing node are counted. Further, the computing node with the largest number of high-heat image files is used as the node to be transferred out, and the computing node with the least number of high-heat image files is used as the node to be transferred in. A connection is established between the node to be transferred out and the node to be transferred in, and the node to be transferred out transfers at least one high-heat image file stored locally to the node to be transferred in for storage, and the node to be transferred in transfers the same number of low-heat image files stored locally to the node to be transferred out for storage.

[0097] By transferring high-heat image files and low-heat image files between computing nodes with higher access load and computing nodes with lower access load, the access load of the two nodes is balanced, further improving the overall image download efficiency of all computing nodes in the resource pool.

[0098] As an implementation manner of the present application, storing the target image file in the new storage node includes:

[0099] Establishing a point-to-point communication link between all storage nodes currently storing the target image file and all new storage nodes;

[0100] Based on the point-to-point communication link, the target image file is transmitted to all new storage nodes through all storage nodes currently storing the target image file, so that each new storage node receives different image slices of the target image file in parallel.

[0101] In one embodiment, the target image file is synchronized between various computing nodes by point-to-point communication. Specifically, the information of all storage nodes currently storing the target image file is obtained through the P2P server, and based on the information of these old storage nodes, a point-to-point communication link is established between the old storage node and the new storage node, and the image fragments of the locally stored target image file are sent to the new storage node through all the old storage nodes, so that each new storage node obtains different parts of the target image file in parallel, thereby improving the efficiency of dynamic scheduling of image files in the resource pool.

[0102] As an implementation manner of the present application, in the process of providing the target image file for download, the process further includes:

[0103] When any computing node receives a download request for the target image file sent by the client, searching for information of all computing nodes storing the target image file in the resource pool according to information of the target image file;

[0104] Based on the information of each computing node storing the target image file, a point-to-point communication link is established between the client and each computing node, so that the client can download different image fragments of the target image file from each computing node in parallel.

[0105] In one embodiment, a peer-to-peer (P2P) method is used for multi-threaded downloading to improve download efficiency. The peer-to-peer download technology can give full play to the advantages of image file fragmentation, distributed storage and redundant images, and fully utilize the upper limit of the throughput performance of each computing node and data disk to achieve the purpose of high-speed transmission. P2P technology is a communication protocol for sharing files in a peer-to-peer network. When downloading images through peer-to-peer, the client first parses the seed file to obtain the tracking address, and further connects to the peer-to-peer server. The peer-to-peer server searches and provides the addresses of all storage nodes of the image file, establishes peer-to-peer communication with the client, and realizes parallel downloading.

[0106] Due to the decentralized nature of peer-to-peer communication, downloaders in the local area network can not only receive file fragments, but also provide their own file fragments to other downloaders. In this way, users can download more threads in parallel when downloading files. Therefore, in the case of multiple users downloading concurrently, each client is also connected to each other and exchanges data blocks that the other party does not have, further improving the overall download efficiency.

[0107] Compared with the existing method of using a separate storage server to manage images, this application can save the disk space required for image storage and reduce the cost of building a cloud platform. At the same time, by dynamically scheduling the number of storage copies of image files to adapt to the download needs of users in different time periods, the overall download performance of the cloud platform is improved. In addition, by introducing the P2P download method, the throughput performance of each computing node and data disk can be fully utilized, providing a faster download rate, achieving the purpose of reducing costs and increasing speed.

[0108] Based on the same inventive concept, an embodiment of the present application provides a mirror storage scheduling system based on resource heat. Figure 3 , Figure 3 FIG. 1 is a schematic diagram of a mirror storage scheduling system based on resource heat proposed in an embodiment of the present application. Figure 3 As shown, the system includes: a cloud platform base, a point-to-point server and a management platform;

[0109] The cloud platform base includes a plurality of computing nodes as in the method provided in the above embodiment, and a disk space for storing a target image file is provided in a data disk of each computing node;

[0110] The point-to-point server is used to collect statistics on the information of the image files stored in each computing node; and to provide information of the corresponding computing nodes for establishing a point-to-point communication link in the resource pool;

[0111] The management platform is used to execute the steps of the method provided in the above embodiment, manage the computing nodes and image files in the resource pool, specify the storage node of each newly created image file, and dynamically schedule the number of copies of each image file stored in the resource pool according to the download heat value of each image file.

[0112] In this embodiment, the system includes a cloud platform base, a peer-to-peer server (P2P server) and a management platform. Among them, the cloud platform base includes a resource pool composed of multiple computing nodes, and each computing node communicates in a peer-to-peer manner. The cloud platform base is not equipped with a dedicated storage server, but a part of the storage space is divided on the data disk of each computing node to store the image fragments of the user's image file, and provide download services to the outside. Users create private images on the cloud platform by uploading and creating cloud hosts, and divide the image files into multiple image fragments through the management platform and store them on the data disks on multiple computing nodes.

[0113] The management platform is used to manage image instances in the resource pool (resource node), execute image sharding algorithms, dynamically schedule image storage, and other operations.

[0114] In the system, a point-to-point proxy service is deployed on each computing node, and communication with the point-to-point server is carried out through the point-to-point proxy service. The point-to-point server continuously collects information about locally stored image files uploaded by each computing node, as well as status information of each computing node. The computing nodes in the resource pool communicate with each other in a point-to-point manner. When a client requests a computing node to download an image file, the point-to-point server obtains information about all computing nodes storing the image file, establishes a point-to-point communication link between the client and each computing node, and then downloads different parts of the image file from each computing node at the same time. The management platform dynamically adjusts the number of images stored in the resource pool according to the download heat value of the image file, and adjusts the distribution of each image file on the computing node, making full use of the IO performance of each disk to improve the user's concurrent download speed and improve the overall performance of the system.

[0115] In addition, given the file sharing feature of peer-to-peer communication, when multiple clients download image files concurrently, each client can also share the image fragments it has downloaded, further improving the efficiency of parallel downloading of image files.

[0116] As an implementation mode of the present application, the management platform is used to select a corresponding number of computing nodes from the resource pool based on a first preset value as storage nodes for the target image file after the target image file is created; control each storage node to store a copy of the target image file respectively; wherein the target image file is stored in each storage node in the form of image slices;

[0117] The peer-to-peer server is used to obtain information about the image files uploaded by each computing node according to a preset time period; the information about the image files includes: the storage location of each image fragment, and the number of downloads of each image file in the previous time period;

[0118] The management platform is also used to obtain the number of downloads of the target image file from the peer-to-peer server during the process of providing the target image file for download, and determine the download heat value of the target image file based on the number of downloads; based on the download heat value, expand or reduce the number of storage copies of the target image file stored in the resource pool.

[0119] As an implementation manner of the present application, the management platform is used to select a corresponding number of computing nodes from the resource pool based on a first preset value as storage nodes for the target image file, specifically including:

[0120] Obtaining the current remaining disk capacity of each computing node in the resource pool, and sorting the computing nodes from large to small according to the remaining disk capacity to obtain a sorting result;

[0121] Based on the first preset value, a plurality of computing nodes at the top of the sorting result are selected as storage nodes for the target image file.

[0122] As an implementation mode of the present application, each computing node includes multiple data disks; the management platform is used to control each storage node to store a copy of the target image file, specifically including:

[0123] According to a preset fragment size, the target image file is split into multiple image fragments;

[0124] For each storage node, all image fragments of the target image file are dispersedly stored in each data disk of the storage node.

[0125] As an implementation mode of the present application, the peer-to-peer server is specifically used to obtain information of all storage nodes storing the target image file according to a preset time period;

[0126] The management platform is used to obtain and parse the image download log of each storage node based on the information of all storage nodes, obtain the download amount of the target image file in the time period, and use the download amount as the download heat value of the target image file.

[0127] As an implementation manner of the present application, the management platform is used to expand or reduce the number of copies of the target image file stored in the resource pool based on the download heat value, specifically including:

[0128] Obtaining a download heat value in the previous time period, and comparing the download heat value with a heat threshold; if the download heat value is greater than the heat threshold, determining that the target image file is of high heat; if the download heat value is less than or equal to the heat threshold, determining that the target image file is of low heat;

[0129] In the case where the target image file is of high popularity, a target number of copies not less than a second preset value is set as the expected number of stored copies of the target image file in the current time period; the second preset value is determined based on the maximum network throughput rate of the resource pool, the maximum transmission rate of the computing node, and the maximum read and write rate of a single data disk; the second preset value is greater than the first preset value;

[0130] When the target image file has low heat, setting the target number of copies of the first preset value as the expected number of stored copies of the target image file in the current time period;

[0131] Based on the expected number of stored copies and the actual number of stored copies of the target image file in the current time period, the number of copies of the target image file stored in the resource pool is expanded or reduced.

[0132] As an implementation manner of the present application, when the download heat value is greater than the heat threshold, the management platform is further used to perform the following steps:

[0133] comparing the target number of copies with a third preset value;

[0134] When the target number of copies is greater than the third preset value, the third preset value is used as the expected number of stored copies of the target image file in the current time period.

[0135] As an implementation mode of the present application, the management platform is used to expand or reduce the number of copies of the target image file stored in the resource pool based on the expected number of stored copies and the actual number of stored copies of the target image file in the current time period, specifically including:

[0136] Calculate the difference between the expected number of stored copies of the target image file and the actual number of stored copies in the current time period;

[0137] When the difference is a positive number, the difference is used as the number of copies that need to be expanded; based on the number of copies that need to be expanded, a corresponding number of computing nodes are selected from the resource pool as new storage nodes, and the target image file is stored in the new storage nodes;

[0138] When the difference is a negative number, the absolute value of the difference is used as the number of copies that need to be reduced; based on the number of copies that need to be reduced, a corresponding number of storage nodes are selected and the target image files in the storage nodes are deleted.

[0139] As an implementation manner of the present application, the management platform is used to store the target image file in the new storage node, specifically including:

[0140] Controlling the peer-to-peer server to obtain information of all storage nodes currently storing the target image file; establishing peer-to-peer communication links between all storage nodes currently storing the target image file and all new storage nodes;

[0141] Based on the point-to-point communication link, all storage nodes currently storing the target image file are controlled to transmit the target image file to all new storage nodes, so that each new storage node receives different image slices of the target image file in parallel.

[0142] As an implementation manner of the present application, the peer-to-peer server is further configured to perform the following steps:

[0143] When any computing node receives a download request for the target image file sent by the client, searching for information of all computing nodes storing the target image file in the resource pool according to information of the target image file;

[0144] Based on the information of each computing node storing the target image file, a point-to-point communication link is established between the client and each computing node, so that the client can download different image fragments of the target image file from each computing node in parallel.

[0145] Regarding the system in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0146] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

[0147] For the method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and components involved are not necessarily required by the present application.

[0148] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the embodiments of the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0149] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0150] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0151] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0152] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they are aware of the basic creative concepts. Therefore, the present application is interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0153] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0154] The above is a detailed introduction to the image storage scheduling method and system based on resource heat provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A mirror storage scheduling method based on resource heat, characterized in that: Applied to a public cloud platform, the resource pool of the public cloud platform includes multiple computing nodes; the method includes: After the target image file is created, a corresponding number of computing nodes are selected from the resource pool based on a first preset value as storage nodes for the target image file; a copy of the target image file is stored in each storage node; wherein the target image file is stored in each storage node in the form of image slices; In the process of providing the target image file for download, the download heat value of the target image file is obtained according to a preset time period; based on the download heat value, the number of copies of the target image file stored in the resource pool is expanded or reduced.

2. The image storage scheduling method based on resource heat according to claim 1 is characterized in that: Selecting a corresponding number of computing nodes from the resource pool based on a first preset value as storage nodes for the target image file includes: Obtaining the current remaining disk capacity of each computing node in the resource pool, and sorting the computing nodes from large to small according to the remaining disk capacity to obtain a sorting result; Based on the first preset value, a plurality of computing nodes at the top of the sorting result are selected as storage nodes for the target image file.

3. The image storage scheduling method based on resource heat according to claim 1 is characterized in that: Each computing node includes multiple data disks; The storing of a copy of the target image file in each storage node includes: According to a preset fragment size, the target image file is split into multiple image fragments; For each storage node, all image fragments of the target image file are dispersedly stored in each data disk of the storage node.

4. The image storage scheduling method based on resource heat according to claim 1 is characterized in that: Obtaining the download heat value of the target image file according to a preset time period includes: According to a preset time period, obtain information of all storage nodes storing the target image file; Based on the information of all storage nodes, the image download log of each storage node is obtained and parsed to obtain the download amount of the target image file in the time period, and the download amount is used as the download heat value of the target image file.

5. The image storage scheduling method based on resource heat according to claim 1 is characterized in that: Based on the download heat value, the number of copies of the target image file stored in the resource pool is expanded or reduced, including: Obtaining a download heat value in the previous time period, and comparing the download heat value with a heat threshold; if the download heat value is greater than the heat threshold, determining that the target image file is of high heat; if the download heat value is less than or equal to the heat threshold, determining that the target image file is of low heat; In the case where the target image file is of high popularity, a target number of copies not less than a second preset value is set as the expected number of stored copies of the target image file in the current time period; the second preset value is determined based on the maximum network throughput rate of the resource pool, the maximum transmission rate of the computing node, and the maximum read and write rate of a single data disk; the second preset value is greater than the first preset value; When the target image file has low heat, setting the target number of copies of the first preset value as the expected number of stored copies of the target image file in the current time period; Based on the expected number of stored copies and the actual number of stored copies of the target image file in the current time period, the number of copies of the target image file stored in the resource pool is expanded or reduced.

6. The image storage scheduling method based on resource heat according to claim 5 is characterized in that: In the case where the download heat value is greater than the heat threshold, the method further includes: comparing the target number of copies with a third preset value; When the target number of copies is greater than the third preset value, the third preset value is used as the expected number of stored copies of the target image file in the current time period.

7. The image storage scheduling method based on resource heat according to claim 1 is characterized in that: Based on the expected number of stored copies and the actual number of stored copies of the target image file in the current time period, the number of copies of the target image file stored in the resource pool is expanded or reduced, including: Calculate the difference between the expected number of stored copies of the target image file and the actual number of stored copies in the current time period; When the difference is a positive number, the difference is used as the number of copies that need to be expanded; based on the number of copies that need to be expanded, a corresponding number of computing nodes are selected from the resource pool as new storage nodes, and the target image file is stored in the new storage nodes; When the difference is a negative number, the absolute value of the difference is used as the number of copies that need to be reduced; based on the number of copies that need to be reduced, a corresponding number of storage nodes are selected and the target image files in the storage nodes are deleted.

8. The image storage scheduling method based on resource heat according to claim 7 is characterized in that: Storing the target image file in the new storage node includes: Establishing a point-to-point communication link between all storage nodes currently storing the target image file and all new storage nodes; Based on the point-to-point communication link, the target image file is transmitted to all new storage nodes through all storage nodes currently storing the target image file, so that each new storage node receives different image slices of the target image file in parallel.

9. The image storage scheduling method based on resource heat according to claim 1, characterized in that: The process of providing the target image file for downloading also includes: When any computing node receives a download request for the target image file sent by the client, searching for information of all computing nodes storing the target image file in the resource pool according to information of the target image file; Based on the information of each computing node storing the target image file, a point-to-point communication link is established between the client and each computing node, so that the client can download different image fragments of the target image file from each computing node in parallel.

10. A mirror storage scheduling system based on resource heat, characterized in that: include: Cloud platform base, peer-to-peer servers and management platform; The cloud platform base includes a plurality of computing nodes according to any one of the methods of claims 1 to 9, and a disk space for storing a target image file is provided in a data disk of each computing node; The point-to-point server is used to collect statistics on the information of the image files stored in each computing node; and to provide information of the corresponding computing nodes for establishing a point-to-point communication link in the resource pool; The management platform is used to execute the steps in the method as described in any one of claims 1 to 9, manage the computing nodes and image files in the resource pool, specify the storage node of each newly created image file, and dynamically schedule the number of copies of each image file stored in the resource pool according to the download heat value of each image file.

Images