A data distribution method, apparatus, device, medium, and program product

Abstract

This invention provides a data distribution method, apparatus, device, medium, and program product, relating to the field of computer technology. The data distribution method includes: obtaining a target disk creation instruction, the target disk creation instruction being used to instruct the download of target data to a target disk storage cluster, the target data including multiple target data blocks; if the target data blocks to be downloaded are stored in a data cache layer, then the target data blocks are downloaded from the data cache layer to the target disk storage cluster; if the target data blocks are not stored in the data cache layer, then the target data blocks are downloaded from a data distribution layer or a data warehouse to the target disk storage cluster. The solution of this invention can minimize access traffic to the data warehouse and reduce access traffic to the data distribution layer, solving the problems of network traffic concentration and single-point distribution capability bottlenecks.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a data distribution method, apparatus, device, medium, and program product. Background Technology

[0002] Both traditional physical servers and cloud-based virtual machine instances mount a disk (called a cloud disk in cloud environments) to store the data required by the operating system; this disk is called the system disk. Meanwhile, most servers and virtual machine instances also mount one or more disks to store application data; these are called data disks. The system disk needs to be pre-configured with the operating system's configuration and data, while the data disk can be created as an empty disk to store data subsequently written by applications, or it can use the same method as the system disk to pre-install data.

[0003] An image is a software package containing the configuration and data required for an operating system to boot and run. It can contain only data from a single system disk, or it can contain data from the system disk and multiple data disks. Its underlying data is typically stored on low-cost cold storage media (such as object storage). When the image is used to create a server or virtual machine instance, the data from the system disk and data disks in the image is imported into the corresponding system disk and data disks mounted on the server or virtual machine instance. Subsequent booting and running of the server and virtual machine instance are independent of the image; data and configuration are directly obtained from the mounted disks.

[0004] Besides the scenarios of creating and starting server and virtual machine instances, another scenario is mounting a new data disk onto an already running server or virtual machine instance. The new data disk can be empty or filled with pre-fabricated data. Pre-fabricated data storage is generally achieved through snapshots, and its data storage and usage are the same as images.

[0005] Therefore, both server or virtual machine instance creation and startup scenarios, as well as system expansion scenarios, can be abstracted as a model of loading image or snapshot data onto the target disk. The existing management method for image and snapshot data involves storing them uniformly in a low-cost storage pool, while cloud disk creation is scheduled across different storage clusters. Data transmission between the image and snapshot data storage pool and the storage cluster hosting the cloud disk occurs over a network, leading to issues of concentrated network traffic and single-point distribution bottlenecks during data distribution. Summary of the Invention

[0006] The purpose of this invention is to provide a data distribution method, apparatus, device, medium, and program product to solve the problems of network traffic concentration and single-point distribution capability bottlenecks in the prior art during data distribution.

[0007] To achieve the above objectives, the present invention is implemented as follows:

[0008] In a first aspect, embodiments of the present invention provide a data distribution method, including:

[0009] Obtain the target disk creation instruction, which instructs the download of target data to the target disk storage cluster, the target data including multiple target data blocks;

[0010] If the data cache layer stores the target data block to be downloaded, then the target data block is downloaded from the data cache layer to the target disk storage cluster;

[0011] If the target data block is not stored in the data cache layer, the target data block is downloaded from the data distribution layer or data warehouse to the target disk storage cluster.

[0012] Optionally, in the data distribution method, downloading the target data block from the data distribution layer or data warehouse to the target disk storage cluster includes:

[0013] If the data distribution layer stores the target data block, then the target data block is downloaded from the data distribution layer to the target disk storage cluster;

[0014] If the data distribution layer does not store the target data block, then the target data block is downloaded from the data warehouse to the target disk storage cluster.

[0015] Optionally, in the data distribution method, the data caching layer includes at least one first storage cluster, each of which is used to download and store full or part of the target data from the data distribution layer.

[0016] The step of downloading the target data block from the data cache layer to the target disk storage cluster includes:

[0017] Based on the current load and traffic information of the data caching layer, a data access routing table is constructed with the goal of load balancing and traffic balancing. The data access routing table is used to indicate the target first storage cluster in the data distribution layer where the target data block is stored.

[0018] Access the routing table based on the data to obtain the target first storage cluster;

[0019] Download the target data block from the target first storage cluster to the target disk storage cluster.

[0020] Optionally, the data distribution method further includes:

[0021] When all target data is downloaded to the target disk storage cluster, the target disk storage cluster is added to the data cache layer.

[0022] Optionally, in the data distribution method, the data distribution layer includes at least one second storage cluster, each of which is used to download and store all or part of the target data from the data warehouse;

[0023] The method further includes:

[0024] If all the target data is downloaded to the target disk storage cluster, the target disk storage cluster is added to the second storage cluster candidate sequence.

[0025] If the number of second storage clusters in the data distribution layer is less than the quantity threshold or the cache period of the target second storage cluster in the data distribution layer has expired, then the target disk storage cluster in the candidate sequence of the second storage clusters is added to the data distribution layer.

[0026] Optionally, in the data distribution method, the data distribution layer includes at least one second storage cluster, each of which is used to download and store all or part of the target data from the data warehouse;

[0027] The method further includes:

[0028] If the target data is downloaded to the target disk storage cluster in full, and the number of second storage clusters in the data distribution layer is less than the quantity threshold or the cache period of the target second storage cluster in the data distribution layer has expired, then the target disk storage cluster is added to the data distribution layer.

[0029] Optionally, in the data distribution method, before downloading the target data block to the target disk storage cluster from the data cache layer if the target data block to be downloaded is stored in the data cache layer, the method further includes:

[0030] Determine whether the target disk storage cluster stores the target data block;

[0031] If the target disk storage cluster does not store the target data block, then it is determined whether the target data block is stored in the data cache layer.

[0032] Optionally, in the data distribution method, before downloading the target data block to the target disk storage cluster from the data cache layer if the target data block to be downloaded is stored in the data cache layer, the method further includes:

[0033] Determine whether the target data is stored in the data warehouse;

[0034] If the target data is stored in the data warehouse, then determine whether the target data block is stored in the data cache layer.

[0035] Secondly, embodiments of the present invention also provide a data distribution apparatus, comprising:

[0036] The acquisition module is used to acquire the target disk creation instruction, which is used to instruct the download of target data to the target disk storage cluster. The target data includes multiple target data blocks.

[0037] The first download module is used to download the target data block to the target disk storage cluster from the data cache layer if the target data block to be downloaded is stored in the data cache layer.

[0038] The second download module is used to download the target data block from the data distribution layer or data warehouse to the target disk storage cluster if the target data block is not stored in the data cache layer.

[0039] Thirdly, embodiments of the present invention also provide a data distribution device, including: a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the data distribution method as described in the first aspect.

[0040] Fourthly, embodiments of the present invention also provide a readable storage medium storing a program, which, when executed by a processor, implements the data distribution method as described in the first aspect.

[0041] Fifthly, embodiments of the present invention also provide a computer program product, including computer instructions, which, when executed by a processor, implement the data distribution method as described in the first aspect.

[0042] The beneficial effects of the above-described technical solution of the present invention are as follows:

[0043] The data distribution method described in this embodiment of the invention obtains a target disk creation instruction, which instructs the download of target data to the target disk storage cluster. The target data includes multiple target data blocks. If the target data blocks to be downloaded are stored in the data cache layer, they are downloaded from the data cache layer to the target disk storage cluster. If the target data blocks are not stored in the data cache layer, they are downloaded from the data distribution layer or the data warehouse to the target disk storage cluster. Thus, through a fully interconnected data cache layer, data block layer, and data warehouse, target data is managed hierarchically, minimizing access traffic to the data warehouse and avoiding the impact on target disk read / write operations caused by bandwidth limitations of the data warehouse and the non-high-speed network between the data warehouse and the target disk storage cluster. Furthermore, it reduces access traffic to the data distribution layer, solving the problems of concentrated network traffic and single-point distribution capability bottlenecks. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the application system architecture of the data distribution method described in an embodiment of the present invention;

[0045] Figure 2 This is a flowchart illustrating the data distribution method described in an embodiment of the present invention;

[0046] Figure 3 This is a flowchart illustrating an implementation of the data distribution method described in this invention.

[0047] Figure 4 This is a schematic diagram of the data distribution device according to an embodiment of the present invention;

[0048] Figure 5 This is a hardware block diagram of the data distribution device described in an embodiment of the present invention. Detailed Implementation

[0049] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0050] The application system architecture of the data distribution method described in this embodiment of the invention is as follows: Figure 1 As shown, it includes at least three layers: a data warehouse, a data distribution layer, and a data cache layer. The data warehouse is used to store the full image data and snapshot data, and therefore can also be called an image registry. The data distribution layer can also be called a distribution group, and the data cache layer can also be called a block cache layer.

[0051] The data warehouse, typically a third-party service, addresses the raw storage of mirrored and snapshot data, providing mirrored storage and shared access capabilities across the entire region. It contains all mirrored and snapshot data. After creation, the initial data is stored, serving as the starting point for subsequent distribution. Both mirrored and snapshot data are initially retrieved from this layer, and identical data only needs to access the data warehouse once during the data distribution layer's construction; subsequent access is restricted to the high-speed interconnected cluster on the storage side.

[0052] The data distribution layer addresses traffic bottlenecks and low-speed network issues in the data warehouse and disk (or cloud disk) storage clusters (i.e., the storage clusters where the disks reside), preventing any impact on disk (or cloud disk) performance. As a data replica of mirrored and snapshot data on the storage side, the data distribution layer includes the full amount of mirrored and snapshot data, used for subsequent caching of mirrored and snapshot data in the storage clusters and distribution of disk (or cloud disk) data. The data distribution layer operates at the cluster level, comprising multiple storage clusters. Each storage cluster stores a full or partial copy of the specified mirrored and snapshot data, with multiple storage clusters constituting the full amount of mirrored and snapshot data. The flow of mirrored and snapshot data on the storage side begins at the data distribution layer. When mirrored and snapshot data are not cached in a particular storage cluster or when the data caching layer has few storage clusters, it handles a portion of the download traffic for mirrored and snapshot data from other storage clusters. Once sufficient mirrored data is cached in the data caching layer, data traffic is distributed across the various storage clusters within the data caching layer, utilizing the concurrent processing capabilities of multiple storage clusters to avoid bandwidth and processing capacity bottlenecks when a single storage cluster provides the entire system traffic.

[0053] It should be noted that the data distribution layer employs a flexible distribution strategy. Each storage cluster in the data distribution layer can either include the full mirrored data, providing data replication capabilities, or it can include a portion of the mirrored data. Access to the full mirrored data is achieved through unified data access scheduling, reducing storage costs and ensuring balanced traffic. The addition and release of storage clusters in the data distribution layer are controlled by multiple dimensions, including the number of storage clusters and their cache expiration periods. When the number of storage clusters in the data distribution layer falls below the maximum member limit, new storage clusters are allowed to be added to distribute access traffic and processing capacity. Once the maximum number of storage clusters is reached, the cache expiration period for each storage cluster is calculated. Storage clusters exceeding their cache expiration period are replaced by the one with the longest cache expiration period when a new storage cluster is added, in reverse order of cache expiration duration. The replaced storage cluster releases its mirrored cached data to minimize the impact on the cluster's saleable space.

[0054] The data caching layer addresses issues such as data access traffic sets, bottlenecks in traffic and processing capacity for single-cluster data distribution, and the reduction of saleable space due to the space occupied by full data caching. As a distributed cache for storage-side image and snapshot data blocks, the data caching layer includes full or partial image and snapshot data for distribution when creating disks (or cloud disks) from image and snapshot data. The data caching layer is also cluster-level. Multiple storage clusters within the data caching layer employ a fully interconnected data distribution mesh structure, with each cluster having equal status. Cached image data can originate from the data distribution layer or other storage clusters within the data caching layer. Storage clusters that cache partial data share image and snapshot data caches. During disk creation, a data access routing table is constructed based on a specified scheduling strategy, selecting storage clusters for traffic and load balancing. Each storage cluster in the data access routing table includes partial cached image and snapshot data. Data for creating cloud disks from images is downloaded concurrently from each storage cluster in the data access routing table to construct the full image data.

[0055] It should also be noted that the joining and leaving of the storage cluster in the data caching layer depends on the creation and release of image data and snapshot data caches. Once a storage cluster has cached all the image data, it can join the data distribution layer.

[0056] Next, the above application system architecture is dynamically constructed, and the specific dynamic construction process is as follows:

[0057] Taking a new image data as an example, there is no cache in the entire application system. At this time, the data distribution layer is empty, and the data download initiated when the image is created is directly downloaded from the data warehouse. At the same time, the downloaded image data will be cached in the current storage cluster, and the current storage cluster will be used as the storage cluster of the data distribution layer to provide a high-speed access channel for subsequent image data access.

[0058] Then, the storage cluster of the data distribution layer will initiate partial data caching operations to each storage cluster through the unified management service role of multiple storage clusters. The mirrored data is evenly scheduled and distributed among the storage clusters for caching. At the same time, the storage clusters with partial cached data are added to the data distribution layer until the number of storage clusters reaches the upper limit. At this time, the data distribution layer is completed.

[0059] Furthermore, the mirrored data continues to be cached in blocks across the storage clusters according to a load balancing algorithm, with each storage cluster having a certain degree of cached data redundancy to achieve traffic distribution for data block access. These storage clusters share data to form a mesh-like data cache layer, and decide whether to apply to join the data distribution layer based on the data caching status. If a storage cluster that has completed caching fails to join the data distribution layer, its cache space will be cleared after a period of retention to release cluster space. This process is then repeated to achieve dynamic balance in the multi-layered data distribution structure.

[0060] To address the issues of concentrated network traffic and single-point distribution capacity bottlenecks in existing data distribution technologies, this invention provides a data distribution method. The method involves obtaining a target disk creation instruction, which directs the download of target data to a target disk storage cluster. The target data includes multiple target data blocks. If the target data blocks to be downloaded are stored in the data cache layer, they are downloaded from the data cache layer to the target disk storage cluster. If the target data blocks are not stored in the data cache layer, they are downloaded from the data distribution layer or the data warehouse to the target disk storage cluster. This implementation method, through a fully interconnected data cache layer, data block layer, and data warehouse, manages the target data in layers, minimizing access traffic to the data warehouse and avoiding the impact on target disk read / write operations caused by bandwidth limitations of the data warehouse and the non-high-speed network between the data warehouse and the target disk storage cluster. Furthermore, it reduces access traffic to the data distribution layer, thus resolving the issues of concentrated network traffic and single-point distribution capacity bottlenecks during data distribution.

[0061] Embodiments of the present invention, such as Figure 2 As shown, a data distribution method is provided, including:

[0062] S201, Obtain the target disk creation instruction. The target disk creation instruction is used to instruct the download of target data to the target disk storage cluster. The target data includes multiple target data blocks.

[0063] It should be noted that, for cloud disks, the target disk creation instruction is a cloud disk creation instruction; for hard disks, the target disk creation instruction is a hard disk creation instruction. The target disk creation instruction is used to instruct the download of the full target data to the target disk storage cluster and to copy the full target data to the target disk. The target data includes image data or snapshot data.

[0064] In one embodiment, optionally, after S201 and before S202, the method further includes:

[0065] Determine whether the target data is stored in the data warehouse;

[0066] If the target data is stored in the data warehouse, then determine whether the target data block is stored in the data cache layer.

[0067] In one embodiment, optionally, after S201 and before S202, the method further includes:

[0068] Determine whether the target disk storage cluster stores the target data block;

[0069] If the target disk storage cluster does not store the target data block, then it is determined whether the target data block is stored in the data cache layer.

[0070] In one embodiment, optionally, after S201 and before S202, the method further includes:

[0071] Determine whether the target data is stored in the data warehouse;

[0072] If the target data is stored in the data warehouse, then determine whether the target disk storage cluster stores the target data block;

[0073] If the target disk storage cluster does not store the target data block, then it is determined whether the target data block is stored in the data cache layer.

[0074] S202, if the data cache layer stores the target data block to be downloaded, then download the target data block from the data cache layer to the target disk storage cluster.

[0075] It is understandable that S202 includes:

[0076] Determine whether the target data block is stored in the data cache layer;

[0077] If the data cache layer stores the target data block to be downloaded, then the target data block is downloaded from the data cache layer to the target disk storage cluster.

[0078] In one embodiment, optionally, the data caching layer includes at least one first storage cluster, each of which is used to download full or part of the target data from the data distribution layer and store it.

[0079] The step of downloading the target data block from the data cache layer to the target disk storage cluster includes:

[0080] Based on the current load and traffic information of the data caching layer, a data access routing table is constructed with the goal of load balancing and traffic balancing. The data access routing table is used to indicate the target first storage cluster in the data distribution layer where the target data block is stored.

[0081] Access the routing table based on the data to obtain the target first storage cluster;

[0082] Download the target data block from the target first storage cluster to the target disk storage cluster.

[0083] In this embodiment of the invention, when downloading the target data block from the data cache layer, it is necessary to first obtain the target first storage cluster where the target data block is stored according to the data access routing table, and then download the target data block from the target first storage cluster to the target disk storage cluster.

[0084] Since the data access routing table is constructed based on the load and traffic information of each first storage cluster in the current data caching layer, and with the goal of load balancing and traffic balancing, the data access routing table can avoid overload or traffic overload of a single first storage cluster, thus achieving the effect of load or traffic balancing.

[0085] Specifically, the data access routing table includes the target first storage cluster corresponding to each target data block.

[0086] It should be noted that the target data includes multiple target data blocks that can be downloaded concurrently.

[0087] S203, if the target data block is not stored in the data cache layer, then download the target data block from the data distribution layer or data warehouse to the target disk storage cluster.

[0088] It is understandable that S202 includes:

[0089] Determine whether the target data block is stored in the data cache layer;

[0090] If the target data block is not stored in the data cache layer, the target data block is downloaded from the data distribution layer or data warehouse to the target disk storage cluster.

[0091] In one implementation, optionally, downloading the target data block from the data distribution layer or data warehouse to the target disk storage cluster includes:

[0092] Determine whether the data distribution layer stores the target data block;

[0093] If the data distribution layer stores the target data block, then the target data block is downloaded from the data distribution layer to the target disk storage cluster;

[0094] If the data distribution layer does not store the target data block, then the target data block is downloaded from the data warehouse to the target disk storage cluster.

[0095] It is understood that the data distribution method described in this embodiment of the invention downloads the target data block from the data warehouse to the target disk storage cluster only when the target data block is not available in the data cache layer and the data distribution layer. This minimizes the access traffic to the data warehouse and avoids the impact on target disk read and write caused by the bandwidth limitations of the data warehouse and the non-high-speed network between the data warehouse and the target disk storage cluster.

[0096] In one embodiment, optionally, the method further includes:

[0097] When all target data is downloaded to the target disk storage cluster, the target disk storage cluster is added to the data cache layer.

[0098] It should be noted that after the target data block is downloaded to the target disk storage cluster, it is determined whether there are any more target data blocks to be downloaded, and then downloaded to the target disk storage cluster according to S202 and S203 described above. If there are no more target data blocks to be downloaded, that is, if the entire target data has been downloaded to the target disk storage cluster, the target disk storage cluster is added to the data cache layer, so that it can serve as a data cache replica to share the load and traffic of subsequent accesses to the same target data, until the cache expires and exits the data cache layer.

[0099] In one embodiment, the data distribution layer may optionally include at least one second storage cluster, each of which is used to download and store all or part of the target data from the data warehouse.

[0100] The method further includes:

[0101] If all the target data is downloaded to the target disk storage cluster, the target disk storage cluster is added to the second storage cluster candidate sequence.

[0102] If the number of second storage clusters in the data distribution layer is less than the quantity threshold or the cache period of the target second storage cluster in the data distribution layer has expired, then the target disk storage cluster in the candidate sequence of the second storage clusters is added to the data distribution layer.

[0103] In this embodiment of the invention, after the target data block is downloaded to the target disk storage cluster, it is determined whether there are any more target data blocks to be downloaded; if there are more target data blocks to be downloaded, they are downloaded to the target disk storage cluster according to the above steps S202 and S203; if there are no target data blocks to be downloaded, it means that all the target data has been downloaded to the target disk storage cluster.

[0104] Then, after all the target data has been downloaded to the target disk storage cluster, the target disk storage cluster is added to the second storage cluster candidate sequence corresponding to the data distribution layer.

[0105] Further, it is determined whether the number of second storage clusters in the data distribution layer is less than a quantity threshold, or whether the cache period of the target second storage cluster in the data distribution layer has expired; if the number of second storage clusters in the data distribution layer is less than the quantity threshold or the cache period of the target second storage cluster in the data distribution layer has expired, then the target disk storage cluster in the candidate sequence of the second storage cluster is added to the data distribution layer, that is, the target disk storage cluster is used as the second storage cluster to share the load and traffic of subsequent access to the same target data.

[0106] It should be noted that after the target disk storage cluster is added to the second storage cluster candidate sequence, if the cache period of the target disk storage cluster has expired and the target disk storage cluster has not been added to the data distribution layer, then the target disk storage cluster is removed from the second storage cluster candidate sequence.

[0107] In one embodiment, optionally, the method further includes:

[0108] If the target data is downloaded to the target disk storage cluster in full, and the number of second storage clusters in the data distribution layer is less than the quantity threshold or the cache period of the target second storage cluster in the data distribution layer has expired, then the target disk storage cluster is added to the data distribution layer.

[0109] In this embodiment of the invention, after the target data block is downloaded to the target disk storage cluster, it is determined whether there are any more target data blocks to be downloaded; if there are more target data blocks to be downloaded, they are downloaded to the target disk storage cluster according to the above steps S202 and S203; if there are no target data blocks to be downloaded, it means that all the target data has been downloaded to the target disk storage cluster.

[0110] Then, if all the target data is downloaded to the target disk storage cluster, determine whether the number of the second storage clusters of the data distribution layer is less than the number threshold, or determine whether the cache period of the target second storage cluster of the data distribution layer has expired.

[0111] If the number of second storage clusters in the data distribution layer is less than the quantity threshold or the cache period of the target second storage cluster in the data distribution layer has expired, then the target disk storage cluster is added to the data distribution layer, that is, the target disk storage cluster is used as the second storage cluster to share the load and traffic of subsequent access to the same target data.

[0112] It should be noted that if the number of second storage clusters in the data distribution layer is greater than or equal to the number threshold and the cache period of the target second storage cluster in the data distribution layer has not expired, then the target disk storage cluster serves as a data cache copy until the cache expires.

[0113] like Figure 3 This is a flowchart illustrating an implementation of the data distribution method according to an embodiment of the present invention. In this embodiment, the data distribution method includes the following steps:

[0114] S301, retrieve cloud disk creation instructions.

[0115] S302, Determine whether mirrored data is stored in the data warehouse.

[0116] If the result of S302 is yes, then proceed to S303 to determine whether there are any more data blocks to be downloaded.

[0117] If the judgment result of S303 is yes, then proceed to S304 and select data block download.

[0118] S305, query data caching layer.

[0119] S306, determine whether the data cache layer stores data blocks.

[0120] If the judgment result of S306 is yes, then proceed to S307, and obtain the storage cluster where the data block in the data cache layer is stored according to the data access routing table.

[0121] S308 downloads data blocks from the storage cluster to the cloud disk storage cluster.

[0122] If the result of S306 is negative, proceed to S309 to determine whether the data distribution layer stores data blocks.

[0123] If the judgment result of S309 is yes, then proceed to S310 to download data blocks from the data distribution layer to the cloud disk storage cluster.

[0124] If the result of S309 is negative, proceed to S311 to download data blocks from the data warehouse to the cloud disk storage cluster.

[0125] S312, determine whether the data block was downloaded successfully.

[0126] If the judgment result of S312 is yes, then proceed to S313 to add the cloud disk storage cluster to the data cache layer.

[0127] If the result of S303 is negative, proceed to S314 to determine whether the data distribution layer needs to be updated.

[0128] If the judgment result of S314 is yes, then proceed to S315 to add the cloud disk storage cluster to the data distribution layer.

[0129] S316 determines whether the cloud disk storage cluster includes the full amount of image data.

[0130] If the result of the judgment in S316 is yes, then proceed to S314.

[0131] In summary, by employing the data distribution method described in this embodiment of the invention, a fully interconnected data distribution network with at least three layers is constructed. In the data caching layer, each storage cluster caches a portion of the data, achieving distributed caching and traffic-balanced caching. This minimizes access traffic to the data warehouse, reduces access traffic to the data distribution layer, avoids network traffic concentration and single-point distribution capacity bottlenecks, and resolves the impact of data warehouse access bandwidth limitations and processing capacity bottlenecks on disk (or cloud disk) read / write operations.

[0132] like Figure 4 As shown, an embodiment of the present invention provides a data distribution device, including:

[0133] The acquisition module 401 is used to acquire a target disk creation instruction, which is used to instruct the download of target data to the target disk storage cluster, and the target data includes multiple target data blocks;

[0134] The first download module 402 is used to download the target data block from the data cache layer to the target disk storage cluster if the target data block to be downloaded is stored in the data cache layer.

[0135] The second download module 403 is used to download the target data block from the data distribution layer or data warehouse to the target disk storage cluster if the target data block is not stored in the data cache layer.

[0136] Optionally, in the data distribution device, the second download module 403 is specifically used for:

[0137] If the data distribution layer stores the target data block, then the target data block is downloaded from the data distribution layer to the target disk storage cluster;

[0138] If the data distribution layer does not store the target data block, then the target data block is downloaded from the data warehouse to the target disk storage cluster.

[0139] Optionally, in the data distribution apparatus, the data caching layer includes at least one first storage cluster, each of the first storage clusters being used to download and store full or part of the target data from the data distribution layer;

[0140] The first download module 402 is specifically used for:

[0141] Based on the current load and traffic information of the data caching layer, a data access routing table is constructed with the goal of load balancing and traffic balancing. The data access routing table is used to indicate the target first storage cluster in the data distribution layer where the target data block is stored.

[0142] Access the routing table based on the data to obtain the target first storage cluster;

[0143] Download the target data block from the target first storage cluster to the target disk storage cluster.

[0144] Optionally, the data distribution apparatus further includes:

[0145] The first adding module is used to add the target disk storage cluster to the data cache layer when the full amount of target data has been downloaded to the target disk storage cluster.

[0146] Optionally, in the data distribution apparatus, the data distribution layer includes at least one second storage cluster, each of the second storage clusters being used to download and store full or part of the target data from the data warehouse;

[0147] The device further includes:

[0148] The second adding module is used to add the target disk storage cluster to the second storage cluster candidate sequence when the full amount of target data is downloaded to the target disk storage cluster.

[0149] If the number of second storage clusters in the data distribution layer is less than the quantity threshold or the cache period of the target second storage cluster in the data distribution layer has expired, then the target disk storage cluster in the candidate sequence of the second storage clusters is added to the data distribution layer.

[0150] Optionally, in the data distribution apparatus, the data distribution layer includes at least one second storage cluster, each of the second storage clusters being used to download and store full or part of the target data from the data warehouse;

[0151] The device further includes:

[0152] The third addition module is used to add the target disk storage cluster to the data distribution layer if the number of second storage clusters in the data distribution layer is less than the number threshold or the cache period of the target second storage cluster in the data distribution layer has expired when all the target data is downloaded to the target disk storage cluster.

[0153] Optionally, the data distribution apparatus further includes:

[0154] The first judgment module is used to determine whether the target disk storage cluster stores the target data block;

[0155] The second judgment module is used to determine whether the target data block is stored in the data cache layer if the target disk storage cluster does not store the target data block.

[0156] Optionally, the data distribution apparatus further includes:

[0157] The third judgment module is used to determine whether the target data is stored in the data warehouse;

[0158] The fourth judgment module is used to determine whether the target data block is stored in the data cache layer if the target data is stored in the data warehouse.

[0159] The data distribution device provided in this embodiment of the invention can execute the above-described data distribution method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0160] like Figure 5 As shown, this embodiment of the invention also provides a data distribution device, including: a processor 501; and a memory 502 connected to the processor 501 via a bus interface, the memory 502 being used to store programs and data used by the processor 501 when performing operations, and the processor 501 calling and executing the programs and data stored in the memory 502.

[0161] The data distribution device further includes a transceiver 503, which is connected to a bus interface and is used to receive and send data under the control of the processor 501.

[0162] Specifically, the transceiver 503 is configured to perform the following processes under the control of the processor 501:

[0163] Obtain the target disk creation instruction, which instructs the download of target data to the target disk storage cluster, the target data including multiple target data blocks;

[0164] The processor 501 is used to read the program and execute the following procedures:

[0165] If the data cache layer stores the target data block to be downloaded, then the target data block is downloaded from the data cache layer to the target disk storage cluster;

[0166] If the target data block is not stored in the data cache layer, the target data block is downloaded from the data distribution layer or data warehouse to the target disk storage cluster.

[0167] Among them, Figure 5 In this context, the bus architecture may include any number of interconnected buses and bridges, specifically linking various circuits together, represented by one or more processors (processor 501) and memory (memory 502). The bus architecture may also link together various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. A bus interface provides a user interface 504. A transceiver 503 may be multiple elements, including transmitters and receivers, providing a unit for communicating with various other devices over a transmission medium. Processor 501 is responsible for managing the bus architecture and general processing, and memory 502 may store data used by processor 501 during operation.

[0168] The processor 501 is responsible for managing the bus architecture and general processing, while the memory 502 can store the data used by the processor 501 when performing operations.

[0169] Optionally, in the data distribution device, the processor 501 is specifically configured to read the program and execute the following processes:

[0170] If the data distribution layer stores the target data block, then the target data block is downloaded from the data distribution layer to the target disk storage cluster;

[0171] If the data distribution layer does not store the target data block, then the target data block is downloaded from the data warehouse to the target disk storage cluster.

[0172] Optionally, in the data distribution device, the data caching layer includes at least one first storage cluster, each of the first storage clusters being used to download and store full or part of the target data from the data distribution layer;

[0173] The processor 501 is specifically used to read the program and execute the following processes:

[0174] Based on the current load and traffic information of the data caching layer, a data access routing table is constructed with the goal of load balancing and traffic balancing. The data access routing table is used to indicate the target first storage cluster in the data distribution layer where the target data block is stored.

[0175] Access the routing table based on the data to obtain the target first storage cluster;

[0176] Download the target data block from the target first storage cluster to the target disk storage cluster.

[0177] Optionally, in the data distribution device, the processor 501 is further configured to read the program and execute the following processes:

[0178] When all target data is downloaded to the target disk storage cluster, the target disk storage cluster is added to the data cache layer.

[0179] Optionally, the data distribution device, wherein the data distribution layer includes at least one second storage cluster, each of the second storage clusters being used to download and store full or part of the target data from the data warehouse;

[0180] The processor 501 is also configured to read the program and execute the following processes:

[0181] If all the target data is downloaded to the target disk storage cluster, the target disk storage cluster is added to the second storage cluster candidate sequence.

[0182] If the number of second storage clusters in the data distribution layer is less than the quantity threshold or the cache period of the target second storage cluster in the data distribution layer has expired, then the target disk storage cluster in the candidate sequence of the second storage clusters is added to the data distribution layer.

[0183] Optionally, the data distribution device, wherein the data distribution layer includes at least one second storage cluster, each of the second storage clusters being used to download and store full or part of the target data from the data warehouse;

[0184] The processor 501 is also configured to read the program and execute the following processes:

[0185] If the target data is downloaded to the target disk storage cluster in full, and the number of second storage clusters in the data distribution layer is less than the quantity threshold or the cache period of the target second storage cluster in the data distribution layer has expired, then the target disk storage cluster is added to the data distribution layer.

[0186] Optionally, in the data distribution device, the processor 501 is further configured to read the program and execute the following processes:

[0187] Determine whether the target disk storage cluster stores the target data block;

[0188] If the target disk storage cluster does not store the target data block, then it is determined whether the target data block is stored in the data cache layer.

[0189] Optionally, in the data distribution device, the processor 501 is further configured to read the program and execute the following processes:

[0190] Determine whether the target data is stored in the data warehouse;

[0191] If the target data is stored in the data warehouse, then determine whether the target data block is stored in the data cache layer.

[0192] A specific embodiment of the present invention also provides a computer-readable storage medium storing a computer program thereon. When the program is executed by a processor, it implements the steps in the above-described data distribution method and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0193] In addition, embodiments of the present invention also provide a computer program product, including computer instructions, which, when executed by a processor, implement the above-described... Figure 2 The various processes of the method embodiments shown can achieve the same technical effect, and will not be described again here to avoid repetition.

[0194] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0195] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can be physically comprised separately, or two or more units can be integrated into one unit. The integrated unit described above can be implemented in hardware or in the form of hardware plus software functional units.

[0196] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions that cause a computer device (which may be a personal computer, server, or network device, etc.) to execute some steps of the transmission and reception methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0197] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A data distribution method, characterized in that, include: Obtain the target disk creation instruction, which instructs the download of target data to the target disk storage cluster, the target data including multiple target data blocks; If the data cache layer stores the target data block to be downloaded, then the target data block is downloaded from the data cache layer to the target disk storage cluster; If the target data block is not stored in the data cache layer, the target data block is downloaded from the data distribution layer or data warehouse to the target disk storage cluster. The data distribution layer includes at least one second storage cluster, each of which is used to download and store all or part of the target data from the data warehouse. The method further includes: If all the target data is downloaded to the target disk storage cluster, the target disk storage cluster is added to the second storage cluster candidate sequence. If the number of second storage clusters in the data distribution layer is less than the quantity threshold or the cache period of the target second storage cluster in the data distribution layer has expired, then the target disk storage cluster in the candidate sequence of the second storage clusters is added to the data distribution layer.

2. The data distribution method according to claim 1, characterized in that, The step of downloading the target data block from the data distribution layer or data warehouse to the target disk storage cluster includes: If the data distribution layer stores the target data block, then the target data block is downloaded from the data distribution layer to the target disk storage cluster; If the data distribution layer does not store the target data block, then the target data block is downloaded from the data warehouse to the target disk storage cluster.

3. The data distribution method according to claim 1, characterized in that, The data caching layer includes at least one first storage cluster, each of which is used to download and store full or part of the target data from the data distribution layer. The step of downloading the target data block from the data cache layer to the target disk storage cluster includes: Based on the current load and traffic information of the data caching layer, a data access routing table is constructed with the goal of load balancing and traffic balancing. The data access routing table is used to indicate the target first storage cluster in the data distribution layer where the target data block is stored. Access the routing table based on the data to obtain the target first storage cluster; Download the target data block from the target first storage cluster to the target disk storage cluster.

4. The data distribution method according to claim 1, characterized in that, The method further includes: When all target data is downloaded to the target disk storage cluster, the target disk storage cluster is added to the data cache layer.

5. The data distribution method according to claim 1, characterized in that, The data distribution layer includes at least one second storage cluster, each of which is used to download and store all or part of the target data from the data warehouse. The method further includes: If the target data is downloaded to the target disk storage cluster in full, and the number of second storage clusters in the data distribution layer is less than the quantity threshold or the cache period of the target second storage cluster in the data distribution layer has expired, then the target disk storage cluster is added to the data distribution layer.

6. The data distribution method according to claim 1, characterized in that, Before downloading the target data block to the target disk storage cluster from the data cache layer, if the target data block to be downloaded is stored in the data cache layer, the method further includes: Determine whether the target disk storage cluster stores the target data block; If the target disk storage cluster does not store the target data block, then it is determined whether the target data block is stored in the data cache layer.

7. The data distribution method according to claim 1, characterized in that, Before downloading the target data block to the target disk storage cluster from the data cache layer, if the target data block to be downloaded is stored in the data cache layer, the method further includes: Determine whether the target data is stored in the data warehouse; If the target data is stored in the data warehouse, then determine whether the target data block is stored in the data cache layer.

8. A data distribution device, characterized in that, include: The acquisition module is used to acquire the target disk creation instruction, which is used to instruct the download of target data to the target disk storage cluster. The target data includes multiple target data blocks. The first download module is used to download the target data block to the target disk storage cluster from the data cache layer if the target data block to be downloaded is stored in the data cache layer. The second download module is used to download the target data block from the data distribution layer or data warehouse to the target disk storage cluster if the target data block is not stored in the data cache layer. The data distribution layer includes at least one second storage cluster, each of which is used to download and store all or part of the target data from the data warehouse. The device further includes: The second adding module is used to add the target disk storage cluster to the second storage cluster candidate sequence when the full amount of target data is downloaded to the target disk storage cluster. If the number of second storage clusters in the data distribution layer is less than the quantity threshold or the cache period of the target second storage cluster in the data distribution layer has expired, then the target disk storage cluster in the candidate sequence of the second storage clusters is added to the data distribution layer.

9. A data distribution device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the data distribution method as described in any one of claims 1 to 7.

10. A readable storage medium, characterized in that, The readable storage medium stores a program that, when executed by a processor, implements the data distribution method as described in any one of claims 1 to 7.

11. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the data distribution method as described in any one of claims 1 to 7.

Images