Data distribution method and device, equipment, medium and program product

Abstract

The invention provides a data distribution method and device, equipment, a medium and a program product, and relates to the technical field of computers. The data distribution method comprises the steps that a target disk creation instruction is obtained, the target disk creation instruction is used for instructing to download target data to a target disk storage cluster, and the target data comprises a plurality of target data blocks; if a target data block to be downloaded is stored in the data cache layer, downloading the target data block from the data cache layer to the target disk storage cluster; and if the target data block is not stored in the data cache layer, downloading the target data block from a data distribution layer or a data warehouse to the target disk storage cluster. According to the scheme, the access flow to the data warehouse can be minimized, the access flow to the data distribution layer is reduced, and the problems of network flow concentration and single-point distribution capability bottleneck are solved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data distribution method, device, equipment, medium and program product. Background Art

[0002] Whether it is a traditional physical server or a virtual machine instance on the cloud, a disk (called a cloud disk in the cloud environment) is mounted to store the data required by the operating system. This disk is called a system disk. At the same time, most servers and virtual machine instances will mount one or more disks for storing application data, which are called data disks. The system disk needs to be pre-configured to load the operating system configuration and data onto the disk, while the data disk can create an empty disk to store the data written by subsequent applications, or it can be pre-embedded with data in the same way as the system.

[0003] An image is a software package that contains the configuration and data required for the startup and operation of an operating system. It can contain data from a single system disk or data from a system disk and multiple data disks. Its underlying data is generally stored on low-cost cold storage media (such as object storage, etc.). When an image is used to create a server or virtual machine instance, the data from the system disk and data disk in the image will be installed into the corresponding system disk and data disk mounted on the server or virtual machine instance. The subsequent startup and operation of the server and virtual machine instance have nothing to do with the image, and data and configuration are directly obtained from the mounted disk.

[0004] In addition to the scenarios of creating and starting servers and virtual machine instances, there is another scenario where a new data disk is mounted on an already running server or virtual machine instance. The data on the new data disk can be an empty disk or filled with pre-made data. The storage of pre-made data is generally achieved through snapshots, and the data storage and use methods are the same as those of images.

[0005] Therefore, whether it is the scenario of server or virtual machine instance creation and startup, or the scenario of system expansion, it can be abstracted into a model of image or snapshot data being loaded into the target disk. The existing management method of image and snapshot data is that the image and snapshot data are uniformly stored in a low-cost storage pool, and the creation of cloud disks is scheduled to different storage clusters. The storage pool of image and snapshot data and the storage cluster where the cloud disk is located are used to transmit data through the network. When distributing data, there are problems such as network traffic concentration and single-point distribution capacity bottlenecks. Summary of the invention

[0006] The technical solution of the present invention aims to provide a data distribution method, device, equipment, medium and program product, which are used to solve the problems of network traffic concentration and single-point distribution capacity bottleneck in data distribution in the prior art.

[0007] To achieve the above object, the present invention is achieved by:

[0008] In a first aspect, an embodiment of the present invention provides a data distribution method, including:

[0009] Obtain a target disk creation instruction, where the target disk creation instruction is used to instruct downloading target data to a target disk storage cluster, where the target data includes a plurality of target data blocks;

[0010] If the data cache layer stores the target data block to be downloaded, downloading the target data block from the data cache layer to the target disk storage cluster;

[0011] If the target data block is not stored in the data cache layer, the target data block is downloaded from the data distribution layer or the data warehouse to the target disk storage cluster.

[0012] Optionally, in the data distribution method, the step of downloading the target data block from the data distribution layer or the data warehouse to the target disk storage cluster comprises:

[0013] If the data distribution layer stores the target data block, downloading the target data block from the data distribution layer to the target disk storage cluster;

[0014] If the data distribution layer does not store the target data block, the target data block is downloaded from the data warehouse to the target disk storage cluster.

[0015] Optionally, in the data distribution method, the data cache layer includes at least one first storage cluster, each of the first storage clusters is used to download and store all or part of the target data from the data distribution layer;

[0016] The downloading of the target data block from the data cache layer to the target disk storage cluster includes:

[0017] According to the current load information and traffic information of the data cache layer, a data access routing table with load balancing and traffic balancing as the goal is constructed, wherein the data access routing table is used to indicate a target first storage cluster storing the target data block in the data distribution layer;

[0018] According to the data access routing table, obtaining the target first storage cluster;

[0019] The target data block is downloaded from the target first storage cluster to the target disk storage cluster.

[0020] Optionally, the data distribution method further comprises:

[0021] When the full amount of target data is downloaded to the target disk storage cluster, the target disk storage cluster is added to the data cache layer.

[0022] Optionally, in the data distribution method, the data distribution layer includes at least one second storage cluster, each of the second storage clusters is used to download and store all or part of the target data from the data warehouse;

[0023] The method further comprises:

[0024] When all target data is downloaded to the target disk storage cluster, adding the target disk storage cluster to the second storage cluster candidate sequence;

[0025] If the number of second storage clusters in the data distribution layer is less than the number threshold or the cache period of the target second storage cluster in the data distribution layer has expired, the target disk storage cluster in the second storage cluster candidate sequence is added to the data distribution layer.

[0026] Optionally, in the data distribution method, the data distribution layer includes at least one second storage cluster, each of the second storage clusters is used to download and store all or part of the target data from the data warehouse;

[0027] The method further comprises:

[0028] When the full amount of target data is downloaded to the target disk storage cluster, if the number of second storage clusters in the data distribution layer is less than the quantity threshold or the cache period of the target second storage cluster of the data distribution layer has expired, the target disk storage cluster is added to the data distribution layer.

[0029] Optionally, in the data distribution method, if the data cache layer stores a target data block to be downloaded, before downloading the target data block from the data cache layer to the target disk storage cluster, the method further comprises:

[0030] Determine whether the target disk storage cluster stores the target data block;

[0031] If the target disk storage cluster does not store the target data block, it is determined whether the target data block is stored in the data cache layer.

[0032] Optionally, in the data distribution method, if the data cache layer stores a target data block to be downloaded, before downloading the target data block from the data cache layer to the target disk storage cluster, the method further comprises:

[0033] Determining whether the target data is stored in the data warehouse;

[0034] If the target data is stored in the data warehouse, it is determined whether the target data block is stored in the data cache layer.

[0035] In a second aspect, an embodiment of the present invention further provides a data distribution device, including:

[0036] An acquisition module, used for acquiring a target disk creation instruction, wherein the target disk creation instruction is used for instructing to download target data to a target disk storage cluster, wherein the target data includes a plurality of target data blocks;

[0037] A first downloading module, configured to download the target data block to be downloaded from the data cache layer to the target disk storage cluster if the target data block to be downloaded is stored in the data cache layer;

[0038] The second downloading module is used to download the target data block from the data distribution layer or the data warehouse to the target disk storage cluster if the target data block is not stored in the data cache layer.

[0039] In a third aspect, an embodiment of the present invention further provides a data distribution device, comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the data distribution method as described in the first aspect is implemented.

[0040] In a fourth aspect, an embodiment of the present invention further provides a readable storage medium, on which a program is stored, and when the program is executed by a processor, the data distribution method as described in the first aspect is implemented.

[0041] In a fifth aspect, an embodiment of the present invention further provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the data distribution method as described in the first aspect.

[0042] The beneficial effects of the above technical solution of the present invention are as follows:

[0043] The data distribution method described in the embodiment of the present invention is adopted to obtain a target disk creation instruction, wherein the target disk creation instruction is used to instruct the downloading of target data to the target disk storage cluster, wherein the target data includes multiple target data blocks; if the target data block to be downloaded is stored in the data cache layer, the target data block is downloaded from the data cache layer to the target disk storage cluster; if the target data block is not stored in the data cache layer, the target data block is downloaded from the data distribution layer or the data warehouse to the target disk storage cluster. In this way, through the fully interconnected data cache layer, data block layer and data warehouse, the target data is managed in layers, the access traffic to the data warehouse is minimized, the impact on the reading and writing of the target disk caused by the bandwidth limitation of the data warehouse and the non-high-speed network between the data warehouse and the target disk storage cluster is avoided, and the access traffic to the data distribution layer is reduced, solving the problems of network traffic concentration and single-point distribution capacity bottlenecks. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A schematic diagram of the application system architecture of the data distribution method according to an embodiment of the present invention;

[0045] Figure 2 A schematic diagram of a flow chart of a data distribution method according to an embodiment of the present invention;

[0046] Figure 3 A schematic diagram of a flow chart of an implementation method of the data distribution method according to an embodiment of the present invention;

[0047] Figure 4 A schematic diagram of the structure of a data distribution device according to an embodiment of the present invention;

[0048] Figure 5 This is a hardware block diagram of the data distribution device described in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0050] The application system architecture of the data distribution method according to the embodiment of the present invention is as follows: Figure 1 As shown, it includes at least three layers, namely a data warehouse, a data distribution layer and a data cache layer. The data warehouse is used to store the full amount of image data and snapshot data, so it can also be called an image warehouse (Image Registry), the data distribution layer can also be called an image distribution group (Distribution Group), and the data cache layer can also be called a data block cache layer (Block Cache).

[0051] The data warehouse is used to solve the original storage problem of mirror data and snapshot data. It is generally a third-party service that provides mirror storage and shared access capabilities for the entire region, including full mirror data and snapshot data. After the mirror data and snapshot data are created, the initial data is stored as the starting point for the subsequent distribution of mirror data and snapshot data. The mirror data and snapshot data in the system are initially obtained from this layer, and the same mirror data and snapshot data only need to access the data warehouse once when building the data distribution layer. Subsequent access is limited to the high-speed interconnection cluster on the storage side.

[0052] The data distribution layer is used to solve the traffic bottleneck and non-high-speed network problems of the data warehouse and disk (or cloud disk) storage cluster (i.e., the storage cluster where the disk is located), and avoid the resulting impact on the performance of the disk (or cloud disk). The data distribution layer is a data copy of the mirror data and snapshot data on the storage side, including the full amount of mirror data and snapshot data, which is used for the subsequent distribution of mirror data and snapshot data cache and disk data (or cloud disk data) in the storage cluster. The data distribution layer is a cluster dimension, including multiple storage clusters, each storage cluster saves the full or partial data copy of the specified mirror data and snapshot data, and multiple storage clusters constitute the full amount of mirror data and snapshot data. The flow of mirror data and snapshot data on the storage side starts from the data distribution layer. When the mirror data and snapshot data are not cached in a certain storage cluster or there are fewer storage clusters in the data cache layer, it is responsible for part of the traffic of mirror data and snapshot data downloads in other storage clusters. When the mirror data has enough cache in the data cache layer, the data traffic is scattered among the storage clusters in the data cache layer, and the concurrent processing capabilities of multiple storage clusters are used to avoid the bottleneck of bandwidth and processing power when a single storage cluster provides the entire system traffic.

[0053] It should be noted that the data distribution layer adopts a flexible distribution strategy. Each storage cluster of the data distribution layer can include the full amount of mirror data and provide data copy capabilities, or it can include partial mirror data. The full amount of mirror data can be accessed through unified scheduling of data access, thereby reducing the storage cost of the storage cluster and ensuring traffic balance. The addition and release of storage clusters at the data distribution layer are controlled by multiple dimensions such as the number of storage clusters and the cache period of the storage clusters. When the number of storage clusters at the data distribution layer does not reach the upper limit of the number of members, new storage clusters are allowed to be added from the perspective of sharing access traffic and processing power. After the number of storage clusters reaches the upper limit, the cache period of each storage cluster will be calculated. When a new storage cluster joins, the storage cluster that exceeds the cache period will be replaced in reverse order according to the cache period length. The replaced storage cluster releases the mirror cache data to reduce the impact on the cluster sales space.

[0054] The data cache layer is used to solve the problem of data access traffic set, the traffic and processing capacity bottleneck problem of single cluster data distribution, and the problem of full data cache occupying cluster space and reducing the available space. The data cache layer is a distributed cache of image data blocks and snapshot data blocks on the storage side, including full or partial image data and snapshot data, which is used for the distribution of data when the subsequent image data and snapshot data create disks (or cloud disks). The data cache layer is also a cluster dimension. A fully interconnected data distribution mesh structure is adopted between multiple storage clusters in the data cache layer. Each storage cluster has an equal status. The cached image data can come from the data distribution layer or from other storage clusters in the data cache layer. The storage clusters that cache partial data share the image data and snapshot data cache. When creating a disk, the storage cluster is selected according to the specified scheduling strategy to build a data access routing table with traffic balancing and load balancing as the goal. The storage clusters in the data access routing table each include partial cache data of the image data and snapshot data. The data for creating a cloud disk by mirroring will be downloaded concurrently from each storage cluster in the data access routing table to build the full image data.

[0055] It should also be noted that the storage cluster joining and exiting the data cache layer is determined by the creation and release of mirror data and snapshot data cache. After a storage cluster has cached the full amount of mirror data, it can join the data distribution layer.

[0056] Next, the above application system architecture is dynamically constructed. The dynamic construction process is as follows:

[0057] Take a new image data as an example. There is no cache in the entire application system. At this time, the data distribution layer is empty. The data download initiated when the image is created is directly downloaded from the data warehouse. At the same time, the downloaded image data will be cached in the current storage cluster and the current storage cluster will be used as the storage cluster of the data distribution layer to provide a high-speed link access channel for subsequent image data access.

[0058] Then, the storage cluster of the data distribution layer will initiate partial data caching operations to each storage cluster through the unified management service role of multiple storage clusters. The mirrored data will be evenly scheduled among the storage clusters to break up the cache. At the same time, the storage cluster with partial cached data will be added to the data distribution layer until the number of storage clusters reaches the upper limit. At this time, the data distribution layer is completed.

[0059] Furthermore, the mirrored data continues to be cached in blocks between the storage clusters according to the balancing algorithm. There is a certain amount of cache data redundancy between the storage clusters to achieve the ability to disperse the traffic of data block access. These storage clusters share data with each other to form a mesh data cache layer, and decide whether to apply to join the data distribution layer based on the data cache situation. If the storage cluster that has completed the cache fails to join the data distribution layer, it will clear the cache space after a period of retention to free up cluster space. This process is repeated to achieve a dynamic balance of the multi-layer data distribution structure.

[0060] In order to solve the problems of network traffic concentration and single-point distribution capacity bottleneck in data distribution in the prior art, an embodiment of the present invention provides a data distribution method, which obtains a target disk creation instruction, wherein the target disk creation instruction is used to instruct the downloading of target data to a target disk storage cluster, and the target data includes multiple target data blocks; if the target data block to be downloaded is stored in the data cache layer, the target data block is downloaded from the data cache layer to the target disk storage cluster; if the target data block is not stored in the data cache layer, the target data block is downloaded from the data distribution layer or the data warehouse to the target disk storage cluster. Therefore, by adopting this implementation method, the target data is hierarchically managed through the fully interconnected data cache layer, data block layer and data warehouse, the access traffic to the data warehouse is minimized, the impact on the reading and writing of the target disk caused by the bandwidth limitation of the data warehouse and the non-high-speed network between the data warehouse and the target disk storage cluster is avoided, and the access traffic to the data distribution layer is reduced, thereby solving the problems of network traffic concentration and single-point distribution capacity bottleneck in data distribution.

[0061] The embodiments of the present invention, as Figure 2 As shown, a data distribution method is provided, comprising:

[0062] S201, obtaining a target disk creation instruction, wherein the target disk creation instruction is used to instruct downloading target data to a target disk storage cluster, wherein the target data includes a plurality of target data blocks.

[0063] It should be noted that, for a cloud disk, the target disk creation instruction is a cloud disk creation instruction; for a disk, the target disk creation instruction is a disk creation instruction. The target disk creation instruction is used to instruct to download the full amount of target data to the target disk storage cluster, and copy the full amount of target data to the target disk. The target data includes mirror data or snapshot data.

[0064] In one implementation manner, optionally, after S201 and before S202, the method further includes:

[0065] Determining whether the target data is stored in the data warehouse;

[0066] If the target data is stored in the data warehouse, it is determined whether the target data block is stored in the data cache layer.

[0067] In one implementation manner, optionally, after S201 and before S202, the method further includes:

[0068] Determine whether the target disk storage cluster stores the target data block;

[0069] If the target disk storage cluster does not store the target data block, it is determined whether the target data block is stored in the data cache layer.

[0070] In one implementation manner, optionally, after S201 and before S202, the method further includes:

[0071] Determining whether the target data is stored in the data warehouse;

[0072] If the target data is stored in the data warehouse, determining whether the target disk storage cluster stores the target data block;

[0073] If the target disk storage cluster does not store the target data block, it is determined whether the target data block is stored in the data cache layer.

[0074] S202: If the data cache layer stores a target data block to be downloaded, download the target data block from the data cache layer to the target disk storage cluster.

[0075] It is understood that S202 includes:

[0076] Determine whether the target data block is stored in the data cache layer;

[0077] If the data cache layer stores the target data block to be downloaded, the target data block is downloaded from the data cache layer to the target disk storage cluster.

[0078] In one embodiment, optionally, the data cache layer includes at least one first storage cluster, each of which is used to download all or part of the target data from the data distribution layer and store it;

[0079] The downloading of the target data block from the data cache layer to the target disk storage cluster includes:

[0080] According to the current load information and traffic information of the data cache layer, a data access routing table with load balancing and traffic balancing as the goal is constructed, wherein the data access routing table is used to indicate a target first storage cluster storing the target data block in the data distribution layer;

[0081] According to the data access routing table, obtaining the target first storage cluster;

[0082] The target data block is downloaded from the target first storage cluster to the target disk storage cluster.

[0083] In an embodiment of the present invention, when downloading the target data block from the data cache layer, it is necessary to first obtain the target first storage cluster where the target data block is stored according to the data access routing table, and then download the target data block from the target first storage cluster to the target disk storage cluster.

[0084] Since the data access routing table is constructed based on the current load information and traffic information of each first storage cluster of the data cache layer and with load balancing and traffic balancing as the goals, the data access routing table can avoid load or traffic overload of a single first storage cluster and achieve load or traffic balancing.

[0085] Specifically, the data access routing table includes a target first storage cluster corresponding to each target data block.

[0086] It should be noted that the multiple target data blocks included in the target data can be downloaded concurrently.

[0087] S203: If the target data block is not stored in the data cache layer, download the target data block from the data distribution layer or the data warehouse to the target disk storage cluster.

[0088] It is understood that S202 includes:

[0089] Determine whether the target data block is stored in the data cache layer;

[0090] If the target data block is not stored in the data cache layer, the target data block is downloaded from the data distribution layer or the data warehouse to the target disk storage cluster.

[0091] In one implementation manner, optionally, downloading the target data block from the data distribution layer or the data warehouse to the target disk storage cluster includes:

[0092] Determine whether the data distribution layer stores the target data block;

[0093] If the data distribution layer stores the target data block, downloading the target data block from the data distribution layer to the target disk storage cluster;

[0094] If the data distribution layer does not store the target data block, the target data block is downloaded from the data warehouse to the target disk storage cluster.

[0095] It can be understood that the data distribution method described in the embodiment of the present invention downloads the target data block from the data warehouse to the target disk storage cluster only when the target data block is not available in the data cache layer and the data distribution layer, thereby minimizing the access traffic to the data warehouse and avoiding the impact on the reading and writing of the target disk due to the bandwidth limitation of the data warehouse and the non-high-speed network between the data warehouse and the target disk storage cluster.

[0096] In one embodiment, optionally, the method further comprises:

[0097] When the full amount of target data is downloaded to the target disk storage cluster, the target disk storage cluster is added to the data cache layer.

[0098] It should be noted that after the target data block is downloaded to the target disk storage cluster, it is determined whether there are target data blocks to be downloaded, and they are downloaded to the target disk storage cluster according to the above S202 and S203. If there are no target data blocks to be downloaded, that is, when the full amount of target data is downloaded to the target disk storage cluster, the target disk storage cluster is added to the data cache layer, so that it can share the load and traffic of subsequent access to the same target data as a data cache copy, until the cache fails and exits the data cache layer.

[0099] In one embodiment, optionally, the data distribution layer includes at least one second storage cluster, each of which is used to download all or part of the target data from the data warehouse and store it;

[0100] The method further comprises:

[0101] When all target data is downloaded to the target disk storage cluster, adding the target disk storage cluster to the second storage cluster candidate sequence;

[0102] If the number of second storage clusters in the data distribution layer is less than the number threshold or the cache period of the target second storage cluster in the data distribution layer has expired, the target disk storage cluster in the second storage cluster candidate sequence is added to the data distribution layer.

[0103] In an embodiment of the present invention, after the target data blocks are downloaded to the target disk storage cluster, it is determined whether there are any target data blocks to be downloaded; if there are any target data blocks to be downloaded, they are downloaded to the target disk storage cluster according to S202 and S203 above; if there are no target data blocks to be downloaded, it means that the full amount of target data has been downloaded to the target disk storage cluster.

[0104] Then, when the full amount of target data is downloaded to the target disk storage cluster, the target disk storage cluster is added to the second storage cluster candidate sequence corresponding to the data distribution layer.

[0105] Further, determine whether the number of second storage clusters of the data distribution layer is less than the number threshold, or determine whether the cache period of the target second storage cluster of the data distribution layer has expired; if the number of second storage clusters of the data distribution layer is less than the number threshold or the cache period of the target second storage cluster of the data distribution layer has expired, then add the target disk storage cluster in the candidate sequence of the second storage cluster to the data distribution layer, that is, use the target disk storage cluster as the second storage cluster to share the load and traffic of subsequent access to the same target data.

[0106] It should be noted that after the target disk storage cluster is added to the second storage cluster candidate sequence, if the cache period of the target disk storage cluster has expired and the target disk storage cluster is added to the data distribution layer, the target disk storage cluster is deleted from the second storage cluster candidate sequence.

[0107] In one embodiment, optionally, the method further comprises:

[0108] When the full amount of target data is downloaded to the target disk storage cluster, if the number of second storage clusters in the data distribution layer is less than the quantity threshold or the cache period of the target second storage cluster of the data distribution layer has expired, the target disk storage cluster is added to the data distribution layer.

[0109] In an embodiment of the present invention, after the target data blocks are downloaded to the target disk storage cluster, it is determined whether there are any target data blocks to be downloaded; if there are any target data blocks to be downloaded, they are downloaded to the target disk storage cluster according to S202 and S203 above; if there are no target data blocks to be downloaded, it means that the full amount of target data has been downloaded to the target disk storage cluster.

[0110] Then, when the full amount of target data is downloaded to the target disk storage cluster, it is determined whether the number of second storage clusters of the data distribution layer is less than the number threshold, or it is determined whether the cache period of the target second storage cluster of the data distribution layer has expired;

[0111] If the number of second storage clusters in the data distribution layer is less than the quantity threshold or the cache period of the target second storage cluster of the data distribution layer has expired, the target disk storage cluster is added to the data distribution layer, that is, the target disk storage cluster is used as the second storage cluster to share the load and traffic of subsequent access to the same target data.

[0112] It should be noted that if the number of second storage clusters in the data distribution layer is greater than or equal to the number threshold and the cache period of the target second storage cluster in the data distribution layer has not expired, the target disk storage cluster serves as a data cache copy until the cache fails.

[0113] like Figure 3 The following is a flow chart of an implementation method of the data distribution method according to an embodiment of the present invention. According to this implementation method, the data distribution method includes the following steps:

[0114] S301, obtaining a cloud disk creation instruction.

[0115] S302: Determine whether mirror data is stored in the data warehouse.

[0116] If the determination result of S302 is yes, the process proceeds to S303 to determine whether there are data blocks to be downloaded.

[0117] If the judgment result of S303 is yes, then enter S304 to select data block download.

[0118] S305, query the data cache layer.

[0119] S306: Determine whether the data cache layer stores data blocks.

[0120] If the judgment result of S306 is yes, then proceed to S307 to obtain the storage cluster where the data block is stored in the data cache layer according to the data access routing table.

[0121] S308, downloading data blocks from the storage cluster to the cloud disk storage cluster.

[0122] If the judgment result of S306 is no, then proceed to S309 to judge whether the data distribution layer stores data blocks.

[0123] If the judgment result of S309 is yes, then enter S310 to download the data block from the data distribution layer to the cloud disk storage cluster.

[0124] If the judgment result of S309 is no, then enter S311 to download the data block from the data warehouse to the cloud disk storage cluster.

[0125] S312, determining whether the data block is downloaded successfully.

[0126] If the judgment result of S312 is yes, then enter S313 to add the cloud disk storage cluster to the data cache layer.

[0127] If the judgment result of S303 is no, then enter S314 to judge whether it is necessary to update the data distribution layer.

[0128] If the judgment result of S314 is yes, then enter S315 to add the cloud disk storage cluster to the data distribution layer.

[0129] S316, determining whether the cloud disk storage cluster includes the full amount of mirror data.

[0130] If the judgment result of S316 is yes, then enter S314.

[0131] In summary, the data distribution method described in the embodiment of the present invention is adopted to build a data distribution network with at least three layers and full interconnection. In the data cache layer, each storage cluster caches part of the data to realize distributed cache and traffic balanced cache, minimize the access traffic to the data warehouse, reduce the access traffic to the data distribution layer, avoid network traffic concentration and single-point distribution capacity bottlenecks, and solve the impact of data warehouse access bandwidth limitations and processing capacity bottlenecks on disk (or cloud disk) reading and writing.

[0132] like Figure 4 As shown, an embodiment of the present invention provides a data distribution device, including:

[0133] An acquisition module 401 is used to acquire a target disk creation instruction, where the target disk creation instruction is used to instruct downloading target data to a target disk storage cluster, where the target data includes multiple target data blocks;

[0134] A first downloading module 402 is configured to download the target data block to be downloaded from the data cache layer to the target disk storage cluster if the target data block to be downloaded is stored in the data cache layer;

[0135] The second downloading module 403 is configured to download the target data block from the data distribution layer or the data warehouse to the target disk storage cluster if the target data block is not stored in the data cache layer.

[0136] Optionally, in the data distribution device, the second download module 403 is specifically used to:

[0137] If the data distribution layer stores the target data block, downloading the target data block from the data distribution layer to the target disk storage cluster;

[0138] If the data distribution layer does not store the target data block, the target data block is downloaded from the data warehouse to the target disk storage cluster.

[0139] Optionally, in the data distribution device, the data cache layer includes at least one first storage cluster, each of the first storage clusters is used to download and store all or part of the target data from the data distribution layer;

[0140] The first download module 402 is specifically used for:

[0141] According to the current load information and traffic information of the data cache layer, a data access routing table with load balancing and traffic balancing as the goal is constructed, wherein the data access routing table is used to indicate a target first storage cluster storing the target data block in the data distribution layer;

[0142] According to the data access routing table, obtaining the target first storage cluster;

[0143] The target data block is downloaded from the target first storage cluster to the target disk storage cluster.

[0144] Optionally, the data distribution device further comprises:

[0145] The first adding module is used to add the target disk storage cluster to the data cache layer when the full amount of target data is downloaded to the target disk storage cluster.

[0146] Optionally, in the data distribution device, the data distribution layer includes at least one second storage cluster, each of the second storage clusters is used to download and store all or part of the target data from the data warehouse;

[0147] The device also includes:

[0148] A second adding module, configured to add the target disk storage cluster to a second storage cluster candidate sequence when all target data are downloaded to the target disk storage cluster;

[0149] If the number of second storage clusters in the data distribution layer is less than the number threshold or the cache period of the target second storage cluster in the data distribution layer has expired, the target disk storage cluster in the second storage cluster candidate sequence is added to the data distribution layer.

[0150] Optionally, in the data distribution device, the data distribution layer includes at least one second storage cluster, each of the second storage clusters is used to download and store all or part of the target data from the data warehouse;

[0151] The device also includes:

[0152] The third adding module is used to add the target disk storage cluster to the data distribution layer when the full amount of target data is downloaded to the target disk storage cluster, if the number of second storage clusters in the data distribution layer is less than the quantity threshold or the cache period of the target second storage cluster of the data distribution layer has expired.

[0153] Optionally, the data distribution device further comprises:

[0154] A first judgment module, used to judge whether the target disk storage cluster stores the target data block;

[0155] The second judgment module is used to judge whether the target data block is stored in the data cache layer if the target disk storage cluster does not store the target data block.

[0156] Optionally, the data distribution device further comprises:

[0157] A third judgment module is used to judge whether the target data is stored in the data warehouse;

[0158] The fourth judgment module is used to judge whether the target data block is stored in the data cache layer if the target data is stored in the data warehouse.

[0159] The data distribution device provided in the embodiment of the present invention can execute the above-mentioned data distribution method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0160] like Figure 5 As shown, an embodiment of the present invention also provides a data distribution device, including: a processor 501; and a memory 502 connected to the processor 501 through a bus interface, the memory 502 is used to store programs and data used by the processor 501 when performing operations, and the processor 501 calls and executes the programs and data stored in the memory 502.

[0161] The data distribution device further includes a transceiver 503, which is connected to the bus interface and is used to receive and send data under the control of the processor 501;

[0162] Specifically, the transceiver 503 is used to perform the following process under the control of the processor 501:

[0163] Obtain a target disk creation instruction, where the target disk creation instruction is used to instruct downloading target data to a target disk storage cluster, where the target data includes a plurality of target data blocks;

[0164] The processor 501 is used to read the program and execute the following process:

[0165] If the data cache layer stores the target data block to be downloaded, downloading the target data block from the data cache layer to the target disk storage cluster;

[0166] If the target data block is not stored in the data cache layer, the target data block is downloaded from the data distribution layer or the data warehouse to the target disk storage cluster.

[0167] Among them, Figure 5 In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically linking together various circuits of one or more processors represented by processor 501 and memory represented by memory 502. The bus architecture may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides a user interface 504. The transceiver 503 may be a plurality of components, i.e., including a transmitter and a receiver, providing a unit for communicating with various other devices on a transmission medium. The processor 501 is responsible for managing the bus architecture and general processing, and the memory 502 may store data used by the processor 501 when performing operations.

[0168] The processor 501 is responsible for managing the bus architecture and general processing, and the memory 502 can store data used by the processor 501 when performing operations.

[0169] Optionally, in the data distribution device, the processor 501 is specifically configured to read the program and execute the following process:

[0170] If the data distribution layer stores the target data block, downloading the target data block from the data distribution layer to the target disk storage cluster;

[0171] If the data distribution layer does not store the target data block, the target data block is downloaded from the data warehouse to the target disk storage cluster.

[0172] Optionally, in the data distribution device, the data cache layer includes at least one first storage cluster, each of the first storage clusters is used to download and store all or part of the target data from the data distribution layer;

[0173] The processor 501 is specifically configured to read the program and execute the following process:

[0174] According to the current load information and traffic information of the data cache layer, a data access routing table with load balancing and traffic balancing as the goal is constructed, wherein the data access routing table is used to indicate a target first storage cluster storing the target data block in the data distribution layer;

[0175] According to the data access routing table, obtaining the target first storage cluster;

[0176] The target data block is downloaded from the target first storage cluster to the target disk storage cluster.

[0177] Optionally, in the data distribution device, the processor 501 is further configured to read the program and execute the following process:

[0178] When the full amount of target data is downloaded to the target disk storage cluster, the target disk storage cluster is added to the data cache layer.

[0179] Optionally, in the data distribution device, the data distribution layer includes at least one second storage cluster, each of the second storage clusters is used to download and store all or part of the target data from the data warehouse;

[0180] The processor 501 is further configured to read the program and execute the following process:

[0181] When all target data is downloaded to the target disk storage cluster, adding the target disk storage cluster to the second storage cluster candidate sequence;

[0182] If the number of second storage clusters in the data distribution layer is less than the number threshold or the cache period of the target second storage cluster in the data distribution layer has expired, the target disk storage cluster in the second storage cluster candidate sequence is added to the data distribution layer.

[0183] Optionally, in the data distribution device, the data distribution layer includes at least one second storage cluster, each of the second storage clusters is used to download and store all or part of the target data from the data warehouse;

[0184] The processor 501 is further configured to read the program and execute the following process:

[0185] When the full amount of target data is downloaded to the target disk storage cluster, if the number of second storage clusters in the data distribution layer is less than the quantity threshold or the cache period of the target second storage cluster of the data distribution layer has expired, the target disk storage cluster is added to the data distribution layer.

[0186] Optionally, in the data distribution device, the processor 501 is further configured to read the program and execute the following process:

[0187] Determine whether the target disk storage cluster stores the target data block;

[0188] If the target disk storage cluster does not store the target data block, it is determined whether the target data block is stored in the data cache layer.

[0189] Optionally, in the data distribution device, the processor 501 is further configured to read the program and execute the following process:

[0190] Determining whether the target data is stored in the data warehouse;

[0191] If the target data is stored in the data warehouse, it is determined whether the target data block is stored in the data cache layer.

[0192] A specific embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the above-mentioned data distribution method are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0193] In addition, an embodiment of the present invention further provides a computer program product, including computer instructions, which, when executed by a processor, implement the above Figure 2 The various processes of the method embodiment shown can achieve the same technical effect, and will not be described again here to avoid repetition.

[0194] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0195] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may be physically included separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0196] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform some steps of the sending and receiving methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0197] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A data distribution method, characterized in that: include: Obtain a target disk creation instruction, where the target disk creation instruction is used to instruct downloading target data to a target disk storage cluster, where the target data includes a plurality of target data blocks; If the data cache layer stores the target data block to be downloaded, downloading the target data block from the data cache layer to the target disk storage cluster; If the target data block is not stored in the data cache layer, the target data block is downloaded from the data distribution layer or the data warehouse to the target disk storage cluster.

2. The data distribution method according to claim 1, characterized in that: The downloading of the target data block from the data distribution layer or the data warehouse to the target disk storage cluster includes: If the data distribution layer stores the target data block, downloading the target data block from the data distribution layer to the target disk storage cluster; If the data distribution layer does not store the target data block, the target data block is downloaded from the data warehouse to the target disk storage cluster.

3. The data distribution method according to claim 1, characterized in that: The data cache layer includes at least one first storage cluster, each of which is used to download and store all or part of the target data from the data distribution layer; The downloading of the target data block from the data cache layer to the target disk storage cluster includes: According to the current load information and traffic information of the data cache layer, a data access routing table with load balancing and traffic balancing as the goal is constructed, wherein the data access routing table is used to indicate a target first storage cluster in the data distribution layer that stores the target data block; According to the data access routing table, obtaining the target first storage cluster; The target data block is downloaded from the target first storage cluster to the target disk storage cluster.

4. The data distribution method according to claim 1, characterized in that: The method further comprises: When the full amount of target data is downloaded to the target disk storage cluster, the target disk storage cluster is added to the data cache layer.

5. The data distribution method according to claim 1, characterized in that: The data distribution layer includes at least one second storage cluster, each of which is used to download and store all or part of the target data from the data warehouse; The method further comprises: When all target data is downloaded to the target disk storage cluster, adding the target disk storage cluster to the second storage cluster candidate sequence; If the number of second storage clusters in the data distribution layer is less than the number threshold or the cache period of the target second storage cluster in the data distribution layer has expired, the target disk storage cluster in the second storage cluster candidate sequence is added to the data distribution layer.

6. The data distribution method according to claim 1, characterized in that: The data distribution layer includes at least one second storage cluster, each of which is used to download and store all or part of the target data from the data warehouse; The method further comprises: When the full amount of target data is downloaded to the target disk storage cluster, if the number of second storage clusters in the data distribution layer is less than the quantity threshold or the cache period of the target second storage cluster of the data distribution layer has expired, the target disk storage cluster is added to the data distribution layer.

7. The data distribution method according to claim 1, characterized in that: If the data cache layer stores the target data block to be downloaded, before downloading the target data block from the data cache layer to the target disk storage cluster, the method further includes: Determine whether the target disk storage cluster stores the target data block; If the target disk storage cluster does not store the target data block, it is determined whether the target data block is stored in the data cache layer.

8. The data distribution method according to claim 1, characterized in that: If the data cache layer stores the target data block to be downloaded, before downloading the target data block from the data cache layer to the target disk storage cluster, the method further includes: Determining whether the target data is stored in the data warehouse; If the target data is stored in the data warehouse, it is determined whether the target data block is stored in the data cache layer.

9. A data distribution device, characterized in that: include: An acquisition module, used for acquiring a target disk creation instruction, wherein the target disk creation instruction is used for instructing to download target data to a target disk storage cluster, wherein the target data includes a plurality of target data blocks; A first downloading module, configured to download the target data block to be downloaded from the data cache layer to the target disk storage cluster if the target data block to be downloaded is stored in the data cache layer; The second downloading module is used to download the target data block from the data distribution layer or the data warehouse to the target disk storage cluster if the target data block is not stored in the data cache layer.

10. A data distribution device, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the data distribution method according to any one of claims 1 to 8 is implemented.

11. A readable storage medium, characterized in that: The readable storage medium stores a program, and when the program is executed by a processor, the data distribution method according to any one of claims 1 to 8 is implemented.

12. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the data distribution method according to any one of claims 1 to 8.

Images