Data storage method and related device

Abstract

Provided in the present application are a data storage method and a related device. The method is applied to a cloud management platform, wherein the cloud management platform is used for managing infrastructure which provides cloud services, and the infrastructure comprises a plurality of distributed storage devices. The method comprises: receiving first request information of a tenant, wherein the first request information is used for acquiring a first file; according to the first request information, determining a first data block set in which the first file is located, wherein the first data block set comprises one or more files, the one or more files have the same attribute, and the first data block set is stored in one storage device; according to the first data block set, determining a first storage device in which the first data block set is located; and reading the first file from the first storage device. By means of the method, at least one file with the same attribute is combined into one data block set and is stored in one storage device, such that only a few storage devices need to be accessed during file retrieval, thereby improving the efficiency of file retrieval.

Description

Data storage method and related equipment

[0001] This application claims priority to the Chinese patent application with application number 202310786193.7 filed with the State Intellectual Property Office of China on June 29, 2023, priority to the Chinese patent application with invention name “A method, device and other equipment for archiving data”, priority to the Chinese patent application with application number 202311368172.X filed with the State Intellectual Property Office of China on October 20, 2023, priority to the Chinese patent application with invention name “A data storage method and related equipment”, all contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of cloud computing, and more specifically, to a data storage method, a data storage device, a computing device cluster, a computer program product, and a computer-readable storage medium. Background Art

[0003] Existing cloud providers all offer data storage services to tenants. For data that tenants require long-term, low-cost storage, cloud providers can provide offline storage devices for storage. These offline storage devices can include non-volatile storage devices such as magnetic tapes and optical disks. However, when a tenant needs to retrieve data, they must load the offline storage device into a drive and seek to a specific location before retrieving the data. Therefore, the latency of data retrieval depends primarily on the sum of the waiting time for the drive to become idle, the time it takes for the robotic arm to load the offline storage device, and the seek time. However, when the amount of data a tenant needs to retrieve is large, the limited number of drives can lead to longer retrieval times and lower efficiency. Furthermore, to ensure data reliability, erasure coding (EC) is a commonly used storage technology. EC technology can split a file into multiple fragments, which are then stored separately on multiple storage devices. When using EC technology for storage, retrieving a file requires invoking multiple drives to access multiple storage devices separately to retrieve all the data in the file, resulting in lower file retrieval efficiency.

[0004] Therefore, how to improve the efficiency of the storage method when retrieving files becomes an urgent problem to be solved.

[0005] Summary of the Invention

[0006] The present application provides a data storage method, a data storage device, a computing device cluster, a computer program product, and a computer-readable storage medium, which can improve the efficiency of the data storage method when retrieving files.

[0007] In a first aspect, a data storage method is provided, which is applied to a cloud management platform. The cloud management platform is used to manage an infrastructure that provides cloud services, and the infrastructure includes a storage device cluster. The storage device cluster includes multiple distributed storage devices. The method includes: receiving a first request message from a tenant, where the first request message is used to obtain a first file; determining, based on the first request message, a first data block set where the first file is located, where the first data block set includes one or more files, where the files included in the first data block set have the same attributes, and where the first data block set is stored in one of the multiple distributed storage devices; determining, based on the first data block set, a first storage device where the first data block set is located; and reading the first file from the first storage device.

[0008] In an embodiment of the present application, the cloud management platform can merge at least one file with the same attributes into a data block set and store the data block set in a single storage device. That is, a tenant's file can be stored in a single storage device as much as possible, and multiple tenant files with the same attributes can be stored in a single storage device as much as possible. Therefore, when retrieving one or more files, fewer storage devices need to be accessed, thereby saving time and improving file retrieval efficiency.

[0009] In combination with the first aspect, in certain implementations of the first aspect, a second request information from a tenant is received, where the second request information is used to obtain multiple files, and the multiple files have the same attributes; based on the second request information, a second data block set where the multiple files are located is determined; based on the second data block set, a second storage device where the second data block set is located is determined; and multiple files are read from the second storage device.

[0010] In an embodiment of the present application, the cloud management platform can merge multiple files with the same attributes into a data block set and store it in a storage device, so that only one storage device needs to be accessed when retrieving the multiple files, thereby improving the efficiency of file retrieval.

[0011] In combination with the first aspect, in certain implementations of the first aspect, configuration information of a tenant is received, the configuration information including attributes of each file in at least one file; based on the configuration information and at least one file, N data block sets are determined, each of the N data block sets includes one or more files in at least one file, the attributes of the files included in each data block set are the same, the N data block sets include a first data block set, and N is a positive integer greater than or equal to 1; the N data block sets are respectively stored in N storage devices in at least one distributed storage device.

[0012] In an embodiment of the present application, the cloud management platform can merge at least one file with the same attributes into a data block set, and store the data block set in a storage device, so that when retrieving the one or more files, only fewer storage devices need to be accessed, thereby saving the time for retrieving files and improving the efficiency of retrieving files.

[0013] In combination with the first aspect, in certain implementations of the first aspect, the attributes of a file include at least one of the following: identification information of the storage bucket to which the file belongs, the directory where the file is located, the storage time of the file, the storage type of the file, the size of the file, and the name of the file, where the storage type of the file is used to indicate the access frequency of the file.

[0014] In an embodiment of the present application, multiple files with the same attributes can be merged into a data block set based on the different attributes of the files, thereby improving the efficiency of retrieving files when the tenant needs to retrieve the files.

[0015] In combination with the first aspect, in certain implementations of the first aspect, at least one file set is determined based on the attributes of each file in at least one file, and each file set includes all files with the same attributes in at least one file; based on the at least one file set, N data block sets are determined, each data block set includes one or more files in a file set in the at least one file set, and the size of each data block set is less than or equal to a first preset threshold.

[0016] In an embodiment of the present application, at least one file of a tenant may be divided into different file sets according to the attributes of the file, thereby merging the files in a file set to determine at least one data block set.

[0017] In combination with the first aspect, in certain implementations of the first aspect, M check data block sets are determined based on N data block sets, and the M check data block sets are used to recover when one or more data block sets in the N data block sets fail, where M is a positive integer greater than or equal to 1; the M check data block sets are stored in M ​​storage devices in at least one distributed storage device, and the M storage devices and the N storage devices are different storage devices.

[0018] In an embodiment of the present application, when storing N data block sets, M check data block sets can also be determined based on the N data block sets, so as to facilitate recovery when some data block sets in the N data block sets fail, thereby improving the reliability of data storage.

[0019] In combination with the first aspect, in certain implementations of the first aspect, each data block set includes P data blocks, each check data block set includes P check data blocks, and based on the p-th data block in each data block set in the N data block sets, the p-th check data block in each check data block set in the M check data block sets is determined, and the size of the p-th data block and / or the p-th check data block is less than or equal to a second preset threshold, p=1,...,P, P is a positive integer greater than or equal to 1.

[0020] In an embodiment of the present application, each data block set in the N data block sets can be divided into at least one data block, and a verification data block of the M verification data block sets can be determined based on a data block of each data block set, thereby determining the M verification data block sets, thereby saving the computing resource overhead and memory overhead when determining the M verification data block sets.

[0021] In combination with the first aspect, in certain implementations of the first aspect, the p-th data block of each data block set in the N data block sets is stored in N storage devices respectively; the p-th check data block of each check data block set in the M check data block sets is stored in M ​​storage devices respectively.

[0022] In an embodiment of the present application, N data block sets and M check data block sets can be stored in N+M storage devices respectively, thereby realizing distributed storage and further improving the reliability of data storage.

[0023] In combination with the first aspect, in some implementations of the first aspect, P data blocks of a data block set are continuously stored in a storage device, and P check data blocks of a check data block set are continuously stored in a storage device.

[0024] In an embodiment of the present application, a set of data blocks can be stored continuously in a storage device, so that when a tenant retrieves a file, it only needs to read the continuous storage space in one storage device to retrieve the file, thereby improving the efficiency of file retrieval.

[0025] In combination with the first aspect, in certain implementations of the first aspect, the N data block sets have the same attributes, the attributes of each data block set are determined according to the attributes of the files included in each data block set, and the N storage devices form a group of storage devices.

[0026] In an embodiment of the present application, a set of N data blocks with the same attributes may be stored in a group of storage devices, thereby improving the efficiency of retrieving multiple files.

[0027] Illustratively, the group of storage devices may be a group of storage devices that are physically close to each other. For example, the group of storage devices may be storage devices located on the same rack. Alternatively, the group of storage devices may be storage devices located at the same location on different racks. Alternatively, the group of storage devices may be a group of storage devices that have similar identification information. For example, when the identification information of the group of storage devices is a number, the identification information of each storage device in the group may be sequentially decremented or incremented.

[0028] In combination with the first aspect, in some implementations of the first aspect, a first mapping relationship is generated, where the first mapping relationship is used to indicate identification information of a data block set where each file is located.

[0029] In an embodiment of the present application, when N data block sets are generated, the correspondence between each data block set and each file can be recorded, thereby facilitating the tenant to retrieve the file.

[0030] In combination with the first aspect, in some implementations of the first aspect, a second mapping relationship is generated, where the second mapping relationship is used to indicate identification information of the storage device where each data block set is located.

[0031] In an embodiment of the present application, when N data block sets are stored in N storage devices, the correspondence between each storage device and each data block set can be recorded, thereby facilitating tenants to retrieve files.

[0032] In combination with the first aspect, in some implementations of the first aspect, the at least one distributed storage device is a non-volatile storage device.

[0033] In the embodiment of the present application, the storage device storing the file or data block set may be a non-volatile storage device, thereby improving the reliability of data storage.

[0034] In a second aspect, a data storage device is provided, which includes a module for implementing the first aspect or any possible implementation of the first aspect.

[0035] In a third aspect, the present application provides a computing device cluster comprising at least one computing device, each computing device comprising a processor and a memory; the processor of at least one computing device is used to execute instructions stored in the memory of at least one computing device, so that the computing device cluster executes the method of the above-mentioned first aspect or any one of the implementation methods of the first aspect.

[0036] In a fourth aspect, the present application provides a computer program product comprising instructions, which, when executed by a computer device cluster, enables the computer device cluster to execute the method of the above-mentioned first aspect or any one of the implementation modes of the first aspect.

[0037] In a fifth aspect, the present application provides a computer-readable storage medium comprising computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method of the above-mentioned first aspect or any one of the implementation methods of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] FIG1 is a schematic structural diagram of a data storage system according to an embodiment of the present application.

[0039] FIG2 is a schematic structural diagram of a data storage system according to another embodiment of the present application.

[0040] FIG3 is a schematic flowchart of a data storage method according to an embodiment of the present application.

[0041] FIG4 is a schematic flowchart of a data storage method according to another embodiment of the present application.

[0042] FIG5 is a schematic diagram of a second graphical interface 500 according to an embodiment of the present application.

[0043] FIG6 is a schematic diagram of determining a data block set according to an embodiment of the present application.

[0044] FIG7 is a schematic flowchart of a data storage method according to another embodiment of the present application.

[0045] FIG8 is a schematic diagram of determining a set of check data blocks according to an embodiment of the present application.

[0046] FIG9 is a schematic diagram of storing N data block sets and M check data block sets in N+M storage devices according to an embodiment of the present application.

[0047] FIG10 is a schematic structural diagram of a data storage device according to an embodiment of the present application.

[0048] FIG11 is a schematic structural diagram of a computing device according to an embodiment of the present application.

[0049] FIG12 is a schematic structural diagram of a computing device cluster according to an embodiment of the present application.

[0050] FIG13 is a schematic diagram of a connection between computing devices 1100A and 1100B via a network according to one embodiment of the present application. DETAILED DESCRIPTION

[0051] The technical solution in this application will be described below with reference to the accompanying drawings.

[0052] The embodiments of the present application will present various aspects, embodiments, or features around a system including multiple devices, components, modules, etc. It should be understood and appreciated that each system may include additional devices, components, modules, etc., and / or may not include all of the devices, components, modules, etc. discussed in conjunction with the figures. Furthermore, combinations of these solutions may also be used.

[0053] Additionally, in the embodiments of this application, words such as "exemplary" and "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" in the embodiments of this application should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of the word "exemplary" is intended to present concepts in a concrete manner.

[0054] The business scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Ordinary technicians in this field will know that with the evolution of technology and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0055] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in yet other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0056] In the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: including the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0057] The methods described in the embodiments of the present application can be applied to various cloud management platforms. The cloud management platform is used to manage the infrastructure that provides cloud services, and the infrastructure includes a storage device cluster. The storage device cluster includes at least one distributed storage device. The at least one distributed storage device can be a non-volatile storage device, or the at least one distributed storage device includes a non-volatile storage medium. The at least one distributed storage device can be, for example, a magnetic tape, an optical disk, etc., which is not limited in the embodiments of the present application.

[0058] FIG1 is a schematic structural diagram of a data storage system provided in an embodiment of the present application. The data storage system 100 in FIG1 may include a cloud management platform 110 and a storage device 121 .

[0059] The cloud management platform 110 is used to manage the infrastructure for providing cloud services, which includes a storage device cluster. The storage device cluster includes a storage device 121, which is a distributed storage device. The cloud management platform 110 can provide at least one cloud service for tenants, which may include, for example, storage services, computing services, etc. The tenant is a tenant of a public cloud who has registered a public cloud account and purchased public cloud resources. After purchasing storage services on the cloud management platform 110, the tenant can upload files or data to be stored to the cloud management platform 110, which will then be stored in the storage device 121. The storage device 121 is a non-volatile storage device. The tenant can also read stored files or data from the cloud management platform 110.

[0060] In some embodiments, the storage device 121 may perform storage based on files. Alternatively, the storage device 121 may perform storage based on objects. That is, the storage device 121 may store at least one file of a tenant.

[0061] In some embodiments, storage device 121 is disposed in storage device library 120. Storage device library 120 may include at least one storage device, such as storage device 121 and / or storage device 122. Storage device library 120 may also correspond to one or more drivers, each of which may receive first indication information from cloud management platform 110. The first indication information is used to instruct the driver to read data or files from storage device 121 and / or storage device 122. The driver may also read data or files from storage device 121 and / or storage device 122 based on the first indication information. When storage device library 120 includes multiple storage devices (such as storage device 121 and storage device 122), storage device library 120 may also include at least one mechanical device (such as a robotic arm). Each mechanical device may be used to determine a storage device to be read (such as storage device 121) from the multiple storage devices in storage device library 120 and place the storage device 121 in the driver.

[0062] For example, the storage device 121 may be a magnetic tape or an optical disc, and the storage device library 120 may be a magnetic tape library or an optical disc library.

[0063] In some embodiments, cloud management platform 110 can manage at least one storage device library, such as storage device library 120 and / or storage device library 130. Storage device library 130 can include at least one storage device, such as storage device 131 and / or storage device 132. Storage device library 130 is similar to storage device library 120 and will not be described in detail here. Storage device 131 and storage device 132 are similar to storage device 121 and will not be described in detail here.

[0064] In some embodiments, the storage devices included in a storage device library may belong to the same storage device cluster. Alternatively, the storage devices included in multiple storage device libraries may belong to the same storage device cluster. Alternatively, at least one storage device managed by the cloud management platform 110 may belong to a storage device cluster. The multiple storage devices in a storage device cluster may be connected directly or via a network. The network may be, for example, a wide area network or a local area network.

[0065] The cloud management platform 110 can receive a first request information from a tenant, where the first request information is used to obtain a first file. The first file can be any one of at least one file stored by the tenant. The cloud management platform 110 can also determine the first data block set where the first file is located based on the first request information. The first data block set includes one or more files, and the attributes of the files included in the first data block set are the same. The first data block set is stored in a storage device among multiple distributed storage devices. The cloud management platform 110 can also determine the first storage device where the first data block set is located based on the first data block set, and read the first file from the first storage device.

[0066] In some embodiments, the cloud management platform 110 may receive a second request from the tenant, the second request being for obtaining multiple files. The multiple files have the same attributes. The cloud management platform 110 may also determine, based on the second request, a second data block set containing the multiple files. That is, the multiple files are located in the same data block set. The cloud management platform 110 may also determine, based on the second data block set, a second storage device containing the second data block set, and read the multiple files from the second storage device.

[0067] In some embodiments, the cloud management platform 110 may receive configuration information from a tenant, where the configuration information includes attributes of each file in at least one file of the tenant. The cloud management platform 110 may also determine N data block sets based on the configuration information and the at least one file. Each of the N data block sets includes one or more files of the tenant, and the files included in each data block set have the same attributes, where N is a positive integer greater than or equal to 1. The cloud management platform 110 may also store the N data block sets in N storage devices of the at least one distributed storage device.

[0068] In some embodiments, the attributes of a file may include at least one of the following: identification information of the storage bucket to which the file belongs, the directory where the file is located, the storage time of the file, the storage type of the file, the size of the file, and the name of the file. The storage type of the file is used to indicate the access frequency of the file.

[0069] In some embodiments, the cloud management platform 110 may determine M parity data block sets based on the N data block sets. The M parity data block sets are used to recover when one or more of the N data block sets fail, where M is a positive integer greater than or equal to 1. The cloud management platform 110 may also store the M parity data block sets in M ​​storage devices of at least one distributed storage device. The M storage devices are different from the N storage devices.

[0070] In some embodiments, the cloud management platform 110 may generate a first mapping relationship. The first mapping relationship may be used to indicate identification information of the data block set where each file is located. The cloud management platform 110 may also generate a second mapping relationship. The second mapping relationship may be used to indicate identification information of the storage device where each data block set is located.

[0071] Figure 2 is a schematic diagram of another data storage system provided in an embodiment of the present application. The data storage system in Figure 2 includes a cloud management platform. The cloud management platform can manage a service access module 210, a metadata management module 220, a distributed management module 230, a cache module 240, and a storage device library 250. The cloud management platform can be, for example, cloud management platform 110 in Figure 1.

[0072] The service access module 210 can provide an object interface and / or a file interface. The object interface can be used to receive at least one object stored by a tenant, each object being stored in the form of a file. The file interface can receive at least one file stored by a tenant. The service access module 210 can also send the at least one file stored by the tenant to the distributed management module 230 and / or the cache module 240. The service access module 210 can also receive metadata for each file stored by the tenant. The service access module 210 can send the metadata for each file to the metadata management module 220.

[0073] The metadata management module 220 manages metadata for each file stored by a tenant. File metadata can be used to describe file attributes. File metadata can include at least one of the following: the identification of the storage bucket to which the file belongs, the directory where the file is located, the file's save time, the file's storage type, the file's size, the file's name, and the file's storage location. The file's storage type indicates how frequently the file is accessed.

[0074] For example, the storage type of a file may include at least one of the following types: standard storage, low-frequency storage, archive storage, and deep archive storage. The file access frequency indicated by standard storage is higher than the file access frequency indicated by low-frequency storage, the file access frequency indicated by low-frequency storage is higher than the file access frequency indicated by archive storage, and the file access frequency indicated by archive storage is higher than the file access frequency indicated by deep archive storage.

[0075] In some embodiments, a tenant can configure the file's storage type when uploading a file. Alternatively, a tenant can configure the file's storage type to be a first storage type when uploading a file. Metadata management module 220 can modify the file's storage type to a second storage type after the file meets a first preset condition. The second storage type is different from the first storage type. The second storage type indicates a lower access frequency than the first storage type. The first preset condition can be determined by metadata management module 220 or configured by the tenant.

[0076] For example, a tenant can directly configure the file's storage type to deep archive when uploading a file. Alternatively, a tenant can configure the file's storage type to standard when uploading a file. The metadata management module 220 can modify the file's storage type to deep archive if the file is cold data with an actual storage age of more than 90 days.

[0077] Cache module 240 can temporarily store at least one file of a tenant. When the data volume of the file cached in cache module 240 is greater than or equal to a third preset threshold, cache module 240 can send the cached file to distributed management module 230. Cache module 240 can include a hard disk drive (HDD) and / or a solid state drive (SSD). The embodiment of the present application does not limit the size of the third preset threshold.

[0078] The distributed management module 230 can perform distributed storage for at least one tenant's file. For example, the distributed management module 230 can generate an erasure code (EC) or a replica based on the at least one file, and then store the at least one file, erasure code, or replica on at least one storage device. The distributed management module 230 can manage at least one storage device library, such as storage device library 250 and / or storage device library 260. Each storage device library can include at least one storage device. For example, storage device library 250 can include storage device 251 and / or storage device 252, and storage device library 260 can include storage device 261 and / or storage device 262.

[0079] In some embodiments, the service access module 210, metadata management module 220, distributed management module 230, and cache module 240 may be integrated into the same device. Alternatively, the service access module 210, metadata management module 220, distributed management module 230, and cache module 240 may be integrated into multiple devices. The embodiments of the present application do not limit the modules included in each of the multiple devices.

[0080] The cloud management platform in Figure 1 or Figure 2 can merge at least one file with the same attributes into a data block set and store the data block set in a single storage device. That is, a tenant's file can be stored in a single storage device as much as possible, and multiple tenant files with the same attributes can be stored in a single storage device as much as possible. Consequently, when retrieving the one or more files, fewer storage devices need to be accessed, thereby saving time and improving file retrieval efficiency.

[0081] Figure 3 is a schematic flow chart of a data storage method provided in an embodiment of the present application. The method in Figure 3 can be executed by the cloud management platform in Figure 1 or Figure 2. The method in Figure 3 includes the following steps.

[0082] Step S310: Receive the first request information of the tenant.

[0083] The cloud management platform may receive a first request message from a tenant, where the first request message is used to obtain a first file, where the first file is any one of at least one file previously stored by the tenant.

[0084] Optionally, the cloud management platform may provide a first graphical interface for the tenant, so that the tenant can request to obtain the first file in the first graphical interface. The embodiment of the present application does not limit the specific content displayed in the first graphical interface.

[0085] Step S320: Determine the first data block set where the first file is located according to the first request information.

[0086] The cloud management platform may determine, based on the first request information, a first data block set in which the first file is located. The first data block set includes one or more files, and the files included in the first data block set have the same attributes. The first data block set is stored in one of the plurality of distributed storage devices.

[0087] Optionally, the cloud management platform may determine the first data block set based on the first mapping relationship and the first request information. The first mapping relationship may be used to indicate identification information of the data block set where each file is located. The first mapping relationship may be described in step S720.

[0088] For example, assuming that the first mapping relationship is expressed in a tabular form (as shown in Table 4 below), the cloud management platform can determine the first data block set corresponding to the first file by querying the first mapping relationship table.

[0089] Step S330: Determine the first storage device where the first data block set is located according to the first data block set.

[0090] Optionally, the cloud management platform can determine the first storage device where the first data block set is located based on the second mapping relationship and the first data block set. The first storage device is one of the multiple distributed storage devices. The second mapping relationship is used to indicate identification information of the storage device where each data block set is located. The second mapping relationship can be described in step S740.

[0091] For example, assuming that the second mapping relationship is expressed in a table form (as shown in Table 5 below), the cloud management platform can determine the first storage device corresponding to the first data block set by querying the second mapping relationship table.

[0092] Step S340: Read the first file from the first storage device.

[0093] After determining the first storage device, the cloud management platform can directly read the first file from the first storage device.

[0094] Optionally, the first mapping relationship may also be used to indicate the offset address of each file in the corresponding data block set and / or the internal storage bucket where each file is located. The second mapping relationship may also be used to indicate the offset address of each data block set in the corresponding storage device. The cloud management platform may determine the storage address of the first data block set in the first storage device and the storage address of the first file in the first data block set based on the first mapping relationship and the second mapping relationship, thereby directly reading the first file from the first storage device.

[0095] Optionally, the cloud management platform may receive a second request information from the tenant, and the second request information is used to obtain multiple files, and the attributes of the multiple files are the same. Since the attributes of the multiple files are the same, the multiple files may be located in the same data block set. The cloud management platform may also determine the second data block set where the multiple files are located based on the second request information. The specific implementation method is similar to step S320 and will not be repeated here. The cloud management platform may also determine the second storage device where the second data block set is located based on the second data block set. The specific implementation method is similar to step S330 and will not be repeated here. The cloud management platform may also read the multiple files directly from the second storage device, and the specific implementation method is similar to step S340 and will not be repeated here.

[0096] In an embodiment of the present application, the cloud management platform can retrieve stored files for a tenant upon request. Because the cloud management platform can merge at least one file with the same attributes into a data block set and store the data block set in a single storage device, a tenant's file can be stored in a single storage device as much as possible, and multiple tenant files with the same attributes can be stored in a single storage device as much as possible. Consequently, when retrieving the one or more files, fewer storage devices need to be accessed, thereby saving time and improving file retrieval efficiency.

[0097] Figure 4 is a schematic flow chart of a data storage method provided by an embodiment of the present application. The method in Figure 4 can be executed by the cloud management platform in Figure 1 or Figure 2. The method in Figure 4 includes the following steps.

[0098] Step S410: Receive tenant configuration information.

[0099] The cloud management platform may receive configuration information of the tenant, where the configuration information includes attributes of each file in the at least one file of the tenant.

[0100] Optionally, the attributes of each file may include at least one of the following: identification information of the storage bucket to which the file belongs, the directory where the file is located, the file's save time, the file's storage type, the file's size, and the file's name. The file's save time may include a first save time and / or a second save time. The first save time is used to indicate the file's storage time, that is, when the file's actual save time is greater than or equal to the first save time, the cloud management platform may delete the file. The file's second save time is used to indicate the file's archiving time, that is, when the file's actual save time is greater than or equal to the second save time, the cloud management platform may set the file's storage type to a preset storage type (e.g., a deep archive storage type). The file's storage type may be used to indicate the file's access frequency.

[0101] When a file belongs to an object storage system, the file name can be used to indicate the directory where the file is located. For example, the file name can be "Directory 1 / File 1", where "Directory 1" is the directory where the file is located.

[0102] Optionally, the cloud management platform may provide a second graphical interface to the tenant, so that the tenant may select or upload the configuration information on the second graphical interface.

[0103] For example, the second graphical interface that the cloud management platform can provide is shown in Figure 5. Figure 5 is a schematic diagram of the second graphical interface 500 provided in an embodiment of the present application. As shown in the second graphical interface 500 in Figure 5, the tenant can select or input the following four items in the second graphical interface 500: identification information of the storage bucket, the name of the file, the first saving time, and the storage type. Among them, the tenant can select or input the identification information of the storage bucket where the file is located in the selection box or input box corresponding to "Identification information of the storage bucket", such as the name or number of the storage bucket where the file is located. The tenant can select or input the name of the file in the selection box or input box corresponding to "Name of the file". The tenant can select or input the first saving time of the file in the selection box or input box corresponding to "First saving time". The tenant can select or input the storage type of the file in the selection box or input box corresponding to "Storage type".

[0104] As shown in Figure 5, the configuration information may include the following content: the first retention time of file 1 located under directory 1 in bucket 1 is 30 days, and the storage type of file 1 is a deep archive storage type; the first retention time of file 2 located under directory 1 in bucket 1 is 30 days, and the storage type of file 2 is a deep archive storage type; the first retention time of file 1 located under directory 2 in bucket 1 is 30 days, and the storage type of file 1 is a deep archive storage type.

[0105] It should be understood that the second graphical interface 500 is merely an example, and the cloud management platform may provide the tenant with a second graphical interface in another form, so that the tenant may select or input configuration information in the second graphical interface in another form.

[0106] Optionally, the cloud management platform may display the second graphical interface 500 to the tenant when the tenant uploads a file, thereby receiving the tenant's configuration information.

[0107] Step S420: Determine N data block sets based on the configuration information and at least one file.

[0108] The cloud management platform may determine N data block sets based on the configuration information and the at least one file. Each of the N data block sets includes one or more files in the at least one file, and the files included in each data block set have the same attributes, where N is a positive integer greater than or equal to 1.

[0109] For example, assuming that the configuration information indicates that File A, File B, and File C have the same attributes, the cloud management platform can determine a data block set based on File A, File B, and File C, as shown in Figure 6. Figure 6 is a schematic diagram of determining a data block set provided by an embodiment of the present application. Figure 6 includes File A, File B, and File C, and the three files have the same attributes. The cloud management platform can directly arrange and merge File A, File B, and File C in sequence to obtain Data Block Set 600.

[0110] Optionally, the size of each data block set may be less than or equal to a first preset threshold. The embodiment of the present application does not limit the size of the first preset threshold, and may be, for example, 1 gigabyte (GB), 10 GB, or the like.

[0111] When determining the data block set, the cloud management platform can divide a file into multiple parts, so that the multiple parts of the file belong to different data block sets respectively, and the size of each data block set is less than or equal to a first preset threshold.

[0112] For example, assuming that the sum of the sizes of file A, file B, and file C is greater than a first preset threshold, the cloud management platform may divide file C into a first part and a second part. The cloud management platform may determine data block set A based on files A, file B, and the first part of file C, and may also determine data block set B based on the second part of file C and other files. The other files are files other than file A, file B, and file C. The sizes of both data block set A and data block set B are less than or equal to the first preset threshold.

[0113] When determining a data block set, the cloud management platform can, based on an allocation algorithm, determine at least one file with the same attributes as at least one data block set of similar size. That is, the cloud management platform can, rather than dividing the file into multiple parts, directly determine, based on the allocation algorithm, N data block sets to the greatest extent possible, with the sizes of these N data block sets falling within a preset range. This embodiment of the present application does not limit this preset range. This embodiment of the present application does not limit the specific implementation of the allocation algorithm.

[0114] Optionally, the cloud management platform may determine at least one file set based on the attributes of each file in the at least one file. Each file set includes all files with the same attributes in the at least one file. The cloud management platform may also determine N data block sets based on the at least one file set. Each of the N data block sets includes one or more files in a file set, and the size of each data block set is less than or equal to a first preset threshold.

[0115] Optionally, when determining the data block set, the cloud management platform may also generate a first mapping relationship. This first mapping relationship may be used to indicate the storage location of each file. The storage location of each file may include identification information of the data block set in which each file is located. The storage location of each file may also include the offset address of each file in the data block set and / or the internal storage bucket in which each file is located.

[0116] Step S430: Store the N data block sets in N storage devices in at least one distributed storage device respectively.

[0117] The cloud management platform can store the determined N data block sets on N storage devices. That is, the cloud management platform can store one data block set on one storage device, and the N data block sets are located on different storage devices. The N storage devices belong to a storage device cluster managed by the cloud management platform, and the storage device cluster includes at least one distributed storage device.

[0118] Optionally, the N storage devices may belong to the same storage device library. Alternatively, at least two of the N storage devices may belong to different storage device libraries. The storage device library may be the storage device library 120 in FIG. 1 , and the storage device may be the storage device 121 in FIG. 1 .

[0119] Optionally, when storing the N data block sets on the N storage devices, the cloud management platform may further generate a second mapping relationship. The second mapping relationship is used to indicate identification information of the storage device where each data block set is located. The second mapping relationship may also be used to indicate an offset address of each data block set on the storage device.

[0120] Optionally, before step S430, the cloud management platform may determine M check data block sets based on the N data block sets. The M check data block sets are used for recovery when one or more of the N data block sets fail, where M is a positive integer greater than or equal to 1. The cloud management platform may also store the M check data block sets in M ​​storage devices of at least one distributed storage device, where the M storage devices are different from the N storage devices.

[0121] The cloud management platform can merge at least one file with the same attributes into a data block set and store this data block set on a single storage device. This means that a tenant's file can be stored on a single storage device, and multiple tenant files with the same attributes can be stored on a single storage device. This reduces the number of storage devices that need to be accessed when retrieving one or more files, saving time and improving efficiency.

[0122] Figure 7 is a schematic flow chart of a data storage method provided in an embodiment of the present application. The method in Figure 7 can be executed by the cloud management platform in Figure 1 or Figure 2. The method in Figure 7 includes the following steps.

[0123] Step S710: Determine at least one file set according to the attribute of each file in at least one file.

[0124] The cloud management platform may determine at least one file set based on the attributes of each file of the tenant. Each file set in the at least one file set may include all files with the same attributes of the at least one file of the tenant.

[0125] Alternatively, the cloud management platform can directly store all files with the same attributes into a single file collection based on each file's attributes. Alternatively, the cloud management platform can categorize and label each file based on its attributes. Files with the same attributes can be assigned the same label. The cloud management platform can also group files with the same label into the same file collection.

[0126] For example, assume that the tenant configuration information is as shown in Table 1 below.

[0127] Table 1 Tenant configuration information table

[0128] Assume that file attributes include the following four items: bucket identification information, the directory where the file is located, and the first save time. Based on the file attributes, each file is classified and labeled, and the classification label information table obtained is shown in Table 2 below.

[0129] Table 2 Classification label information table

[0130] As shown in Table 2, since Files 1, 2, and 4 in Directory 1 in Bucket 1 all belong to the same bucket, directory, and first save time, the tag information for Files 1, 2, and 4 in Directory 1 in Bucket 1 is all Tag 1. Since the first save time for File 3 in Directory 1 in Bucket 1 is different from the first save time for File 1 in Directory 1 in Bucket 1, the tag information for File 3 in Directory 1 in Bucket 1 is Tag 2. Since Files 1, 3, and 4 in Directory 2 in Bucket 1 all belong to the same bucket, directory, and first save time, and the directories for Files 1, 3, and 4 in Directory 2 in Bucket 1 are different from the directories for Files 1 and 3 in Directory 1 in Bucket 1, the tag information for Files 1, 3, and 4 in Directory 2 in Bucket 1 is all Tag 3. Similarly, File 2 in Directory 2 in Bucket 1 has Tag 4, and Files 1, 2, and 3 in Directory 1 in Bucket 2 all have Tag 5.

[0131] Based on the classification tag information shown in Table 2, the cloud management platform can group files with the same tag information into the same file set. For example, File 1, File 2, and File 4 in Directory 1 in Bucket 1 belong to File Set 1. File 3 in Directory 1 in Bucket 1 belongs to File Set 2. File 1, File 3, and File 4 in Directory 2 in Bucket 1 belong to File Set 3. File 2 in Directory 2 in Bucket 1 belongs to File Set 4. File 1, File 2, and File 3 in Directory 1 in Bucket 2 belong to File Set 5.

[0132] Step S720: Determine N data block sets based on at least one file set.

[0133] The cloud management platform may determine N data block sets based on the at least one file set, where N is a positive integer greater than or equal to 1. Each data block set includes one or more files in a file set in the at least one file set, and a size of each data block set is less than or equal to a first preset threshold.

[0134] Optionally, the cloud management platform may sequentially arrange and merge one or more files in a file set to determine one or more data block sets in the N data block sets. FIG6 shows the manner in which the cloud management platform sequentially arranges and merges multiple files. Alternatively, the cloud management platform may allocate one or more files in a file set according to an allocation algorithm to determine one or more data block sets in the N data block sets. The size of each data block set in the N data block sets is less than or equal to a first preset threshold.

[0135] In some embodiments, the cloud management platform may divide a file into multiple parts, such that the multiple parts of the file belong to different data block sets, and the size of each data block set is less than or equal to a first preset threshold. Alternatively, when the cloud management platform determines the data block sets based on the allocation algorithm, the cloud management platform may not divide a file into multiple parts, thereby ensuring that all data of a file can be stored on a single storage device.

[0136] Optionally, the cloud management platform may merge one or more files in a file set into a data block set through data migration, for example, through a multi-segment interface of an object.

[0137] Optionally, the cloud management platform may record the attributes of each of the N data block sets. The attributes of each data block set include at least one of the following: identification information of the storage bucket to which the data block set belongs, identification information of the data block set, size of the data block set, and tag information of the data block set. The identification information of the data block set may include, for example, the name or number of the data block set. The tag information of the data block set may be tag information of a file included in the data block set.

[0138] For example, assume that the attributes of the obtained set of N data blocks are as shown in Table 3.

[0139] Table 3 Attribute table of data block set

[0140] As shown in Table 3, data block set 1 can be stored in internal bucket 1. The size of data block set 1 is 10 GB, and the tag information of data block set 1 is tag 1. Internal bucket 1 is a storage bucket managed by the cloud management platform and may be independent of the tenant. In other words, the tenant may not be aware of internal bucket 1.

[0141] Optionally, the cloud management platform may generate a first mapping relationship that may be used to indicate the storage location of each file. The storage location of each file may include identification information of the data block set in which each file is located. The storage location of each file may also include an offset address of each file in the data block set and / or an internal storage bucket in which each file is located.

[0142] For example, the first mapping relationship may be as shown in Table 4. The storage location of a file may be represented by an array whose format is (internal storage bucket where the file is located, data block set where the file is located, offset address of the file in the data block set). That is, the first data in the storage location of a file represents the internal storage bucket where the file is located, the second data represents the data block set where the file is located, and the third data represents the offset address of the file in the data block set.

[0143] Table 4: First mapping relationship table

[0144] As shown in Table 4, the first mapping relationship can be used to indicate that the storage location of file 1 under directory 1 in bucket 1 is (internal bucket 1, data block set 1, offset address 1). That is, file 1 under directory 1 in bucket 1 is located in data block set 1 in internal bucket 1, and the offset address of file 1 in data block set 1 is offset address 1. Based on this first mapping relationship, the data block set where each file is located and the specific location of each file in the data block set can be determined.

[0145] Step S730: Determine M check data block sets based on the N data block sets.

[0146] The cloud management platform may determine M check data block sets based on the N data block sets. The M check data block sets are used for recovery when one or more data block sets in the N data block sets fail, where M is a positive integer greater than or equal to 1.

[0147] Optionally, the cloud management platform can determine the M check data block sets corresponding to the N data block sets based on EC technology. The embodiment of the present application does not limit the specific EC technology, for example, it may include array erasure code (array code), Reed-Solomon erasure code (Reed-Solomon, RS) and low-density parity check erasure code (low density parity check code, LDPC), etc. Alternatively, the cloud management platform can also determine the M check data block sets corresponding to the N data block sets based on other verification methods, and the embodiment of the present application does not limit the specific verification method. When at most M data in the N data block sets and M check data block sets fail, the cloud management platform can recover the failed data based on other data except the failed data, thereby enhancing the reliability of data storage.

[0148] Optionally, the cloud management platform can directly calculate M check data block sets based on the N data block sets. Alternatively, the cloud management platform can divide each data block set in the N data block sets into P data blocks, and determine the p-th check data block of each check data block set in the M check data block sets based on the p-th data block of each data block set in the N data block sets, thereby determining the M check data block sets, as shown in FIG8 . Each check data block set in the M check data block sets includes P check data blocks. P is a positive integer greater than or equal to 1, p=1, ..., P.

[0149] Figure 8 is a schematic diagram of determining a verification data block set provided by an embodiment of the present application. Figure 8 includes a data block set 810, a data block set 820, and a verification data block set 830. Among them, the data block set 810 and the data block set 820 both include P data blocks, and the P data blocks are the 1st data block to the Pth data block respectively. The cloud management platform can use EC technology to determine the 1st verification data block of the verification data block set 830 based on the 1st data block of the data block set 810 and the 1st data block of the data block set 820. Similarly, the cloud management platform can use EC technology to determine the pth verification data block of the verification data block set 830 based on the pth data block of the data block set 810 and the pth data block of the data block set 820, p=1,...,P.

[0150] The following uses RS technology as an example. Assuming that N is 5 and M is 3, then N+M is 8. The cloud management platform can concatenate the pth data block in each of the five data block sets into a matrix D. When the five data blocks are D1, D2, D3, D4, and D5, D = [D1, D2, D3, D4, and D5]. T . Where T represents transpose. If the number of rows of each of the 5 data blocks is d1 and the number of columns is d2, then the number of rows of the matrix D is 5*d1 and the number of columns is d2. The cloud management platform can also construct a matrix B with 8*d1 rows and d2 columns. Moreover, any 5 row vectors of the matrix B are independent of each other, that is, any 5 row vectors forming a 5*5 matrix are reversible. The cloud management platform can also determine the matrix C based on B*D=C, and calculate it based on C=[D,C1,C2,C3] T , determine the pth check data block in each of the three check data block sets. The three check data blocks are C1, C2, and C3 respectively.

[0151] Exemplarily, the cloud management platform may represent each data block set in a matrix form, or may represent the data blocks of each data block set in a matrix form.

[0152] In some embodiments, the size of the p-th data block in each data block set and / or the p-th check data block in each check data block set is less than or equal to a second preset threshold. The embodiment of the present application does not limit the size of the second preset threshold. For example, the size of the second preset threshold can be 1 mebibyte (MB), 2MB, 5MB, etc.

[0153] In some embodiments, the N data block sets have the same attributes. The attributes of each data block set are determined based on the attributes of the files included in each data block set. The N data blocks are respectively stored in N storage devices, and the N storage devices form a group of storage devices.

[0154] Illustratively, the group of storage devices may be a group of storage devices that are physically close to each other. For example, the group of storage devices may be storage devices located on the same rack. Alternatively, the group of storage devices may be storage devices located at the same location on different racks. Alternatively, the group of storage devices may be a group of storage devices that have similar identification information. For example, when the identification information of the group of storage devices is a number, the identification information of each storage device in the group may be sequentially decremented or incremented.

[0155] In other words, the cloud management platform can determine at least one data block set based on the configuration information and at least one tenant file. The cloud management platform can also store N data block sets with the same attributes in the at least one data block set, respectively, on N storage devices. The cloud management platform can also determine M check data block sets based on the N data block sets with the same attributes, and store the M check data block sets in the M storage devices, respectively.

[0156] Step S740: Store the N data block sets and the M check data block sets in N+M storage devices respectively.

[0157] The cloud management platform may store N data block sets in N storage devices respectively, and may also store M verification data block sets in M ​​storage devices respectively, where the N storage devices and the M storage devices are different devices.

[0158] Optionally, after determining N complete data block sets, the cloud management platform may directly store the N complete data block sets in N storage devices. The cloud management platform may also directly store M complete verification data block sets in M ​​storage devices after determining M complete verification data block sets.

[0159] Optionally, after determining the p-th verification data block of each verification data block set in the M verification data block sets based on the p-th data block of each data block set in the N data block sets, the cloud management platform can store the p-th data block of each data block set in the N data block sets in N storage devices respectively, and can also store the p-th verification data block of each verification data block set in the M verification data block sets in M ​​storage devices respectively, as shown in Figure 9.

[0160] FIG9 is a schematic diagram of storing N data block sets and M check data block sets in N+M storage devices, as provided by an embodiment of the present application. FIG9 includes storage device 910, storage device 920, and storage device 930. After determining the first check data block of check data block set 830 based on the first data block of data block set 810 and the first data block of data block set 820 in FIG8, the cloud management platform may store the first data block of data block set 810 in storage device 910, the first data block of data block set 820 in storage device 920, and the first check data block of check data block set 830 in storage device 930. Similarly, after determining the second check data block of the check data block set 830 based on the second data block of the data block set 810 and the second data block of the data block set 820, the second data block of the data block set 810, the second data block of the data block set 820 and the second check data block of the check data block set 830 can be stored in the storage device 910, the storage device 920 and the storage device 930 respectively.

[0161] Illustratively, at least two of the three storage devices in FIG. 9 may be located in the same or different storage device libraries.

[0162] Exemplarily, after storing the p-th data block of each data block set in the N data block sets and the p-th verification data block of each verification data block set in the M verification data block sets respectively in N+M storage devices, the cloud management platform may continue to determine the p+1-th verification data block of each verification data block set in the M verification data block sets based on the p+1-th data block of each data block set in the N data block sets. The cloud management platform may also store the p+1-th data block of each data block set in the N data block sets and the p+1-th verification data block of each verification data block set in the M verification data block sets respectively in N+M storage devices. Each data block of the n-th data block set in the N data block sets is stored in the n-th storage device in the N storage devices, and each data block of the n-th data block set is continuously stored in the n-th storage device, where n=1,...,N. Each check data block of the mth check data block set among the M check data block sets is stored in the mth storage device among the M storage devices, and each check data block of the mth check data block set is stored continuously in the mth storage device, m=1,...,M.

[0163] Optionally, the P data blocks of a data block set are stored continuously in one storage device, and the P check data blocks of a check data block set are stored continuously in one storage device.

[0164] Optionally, the cloud management platform may generate a second mapping relationship that indicates the storage location of each data block set. The storage location of each data block set may include identification information of the storage device where each data block set is located. The storage location of each data block set may also include an offset address of each data block set in the storage device.

[0165] For example, the second mapping relationship may be as shown in Table 5. The storage location of the data block set may be represented by an array whose format is (storage device where the data block set is located, offset address of the data block set in the storage device). That is, the first data in the storage location of the data block set represents the storage device where the data block set is located, and the second data represents the offset address of the data block set in the storage device.

[0166] Table 5 Second mapping relationship table

[0167] As shown in Table 5, the second mapping relationship can be used to indicate that the storage location of data block set 1 is (storage device 1, offset address 12). That is, data block set 1 is located in storage device 1, and the offset address of data block set 1 in storage device 1 is offset address 12. Based on this second mapping relationship, the storage device where each data block set is located and the specific location of each data block set in the storage device can be determined.

[0168] It should be understood that Tables 1 to 5 are merely exemplary and that other forms (e.g., matrices, arrays, functions, etc.) may be used to represent Tables 1 to 5, and the present application does not limit this. The present application does not limit the specific form of the partial data in Tables 1 to 5. For example, the partial data in Tables 1 to 5 may include at least one character, and the at least one character may include any one or more of a letter, a number, or a symbol. Alternatively, the partial data in Tables 1 to 5 may be represented in the form of a matrix, an array, a function, or the like.

[0169] Optionally, the cloud management platform may generate a third mapping relationship that indicates the storage location of each verification data block set. The storage location of each verification data block set may include identification information of the storage device where each verification data block set is located. The storage location of each verification data block set may also include the offset address of each verification data block set in the storage device. The specific expression of this third mapping relationship is similar to that of the second mapping relationship and will not be repeated here.

[0170] In an embodiment of the present application, the cloud management platform can merge at least one file with the same attributes into a data block set, and store the data block set in a storage device. That is, a tenant's file can be stored in one storage device as much as possible, and multiple tenants' files with the same attributes can be stored in one storage device as much as possible. Therefore, when retrieving the one or more files, the number of storage devices that need to be accessed is relatively small, thereby saving time in retrieving files and improving the efficiency of retrieving files. At the same time, the method of determining the check data blocks of the check data block set based on the data blocks of the data block set can save memory overhead when calculating the check data block set.

[0171] Figure 10 is a schematic structural diagram of a data storage device provided in an embodiment of the present application. The data storage device 1000 in Figure 10 includes a receiving module 1010 and a processing module 1020. The data storage device 1000 in Figure 10 can be applied to a cloud management platform, such as the cloud management platform in Figure 1 or Figure 2.

[0172] The receiving module 1010 may be configured to receive a first request from a tenant, where the first request is for obtaining a first file. The receiving module 1010 may be configured to execute step S310 in FIG. 3 and step S410 in FIG. 4 .

[0173] Processing module 1020 can be configured to determine, based on the first request information, a first data block set where the first file is located. Processing module 1020 can also be configured to determine, based on the first data block set, a first storage device where the first data block set is located, and read the first file from the first storage device. Processing module 1020 can execute steps S320 to S340 in FIG. 3 , steps S420 to S430 in FIG. 4 , and steps S710 to S740 in FIG. 7 .

[0174] The receiving module 1010 and the processing module 1020 can be implemented by software or hardware. For example, the implementation of the receiving module 1010 is described below using the receiving module 1010 as an example. Similarly, the implementation of the processing module 1020 can refer to the implementation of the receiving module 1010.

[0175] As an example of a software functional unit, the receiving module 1010 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the computing instance may be one or more. For example, the receiving module 1010 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple geographically close data centers. Typically, a region may include multiple AZs.

[0176] Similarly, multiple hosts / virtual machines / containers running the code can be distributed within the same VPC or across multiple VPCs. Typically, a VPC is set up within a region. Cross-region communication between two VPCs within the same region, or between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.

[0177] As an example of a hardware functional unit, the receiving module 1010 may include at least one computing device, such as a server. Alternatively, the receiving module 1010 may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0178] The multiple computing devices included in the receiving module 1010 can be distributed in the same region or in different regions. The multiple computing devices included in the receiving module 1010 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the receiving module 1010 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, and other computing devices.

[0179] Therefore, the modules of each example described in the embodiments of this application can be implemented with electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0180] It should be noted that: when the device provided in the above embodiment executes the above method, only the division of the above functional modules is used as an example. In actual application, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. For example, the receiving module 1010 can be used to execute any step in the above method, and the processing module 1020 can be used to execute any step in the above method. The steps that the receiving module 1010 and the processing module 1020 are responsible for implementing can be specified as needed, and the full functions of the above device can be realized by respectively implementing different steps in the above method through the receiving module 1010 and the processing module 1020.

[0181] In addition, the device and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation processes are detailed in the method embodiments above and will not be repeated here.

[0182] The method provided in the embodiment of the present application can be performed by a computing device, which can also be referred to as a computer system. It includes a hardware layer, an operating system layer running on the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as a processing unit, a memory and a memory control unit, and the function and structure of the hardware are subsequently described in detail. The operating system is any one or more computer operating systems that implement business processing through a process, for example, a Linux operating system, a Unix operating system, an Android operating system, an iOS operating system or a Windows operating system. The application layer includes applications such as a browser, an address book, a word processing software, and an instant messaging software. Furthermore, optionally, the computer system is a handheld device such as a smart phone, or a terminal device such as a personal computer, and this application is not particularly limited, as long as the method provided in the embodiment of the present application can be used. The execution subject of the method provided in the embodiment of the present application can be a computing device, or a functional module in a computing device that can call a program and execute a program.

[0183] FIG11 is a schematic block diagram of a computing device 1100 provided in an embodiment of the present application. The computing device 1100 may be a server, a computer, or other device with computing capabilities. The computing device 1100 shown in FIG11 includes at least one processor 1110 and a memory 1120.

[0184] It should be understood that this application does not limit the number of processors and memories in the computing device 1100.

[0185] The processor 1110 executes the instructions in the memory 1120 so that the computing device 1100 implements the method provided in this application. Alternatively, the processor 1110 executes the instructions in the memory 1120 so that the computing device 1100 implements the functional modules provided in this application, thereby implementing the method provided in this application.

[0186] Optionally, the computing device 1100 further includes a communication interface 1130. The communication interface 1130 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 1100 and other devices or a communication network.

[0187] Optionally, the computing device 1100 further includes a system bus 1140, wherein the processor 1110, the memory 1120, and the communication interface 1130 are respectively connected to the system bus 1140. The processor 1110 can access the memory 1120 through the system bus 1140. For example, the processor 1110 can read and write data or execute code in the memory 1120 through the system bus 1140. The system bus 1140 is a peripheral component interconnect express (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus 1140 is divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in Figure 11, but it does not mean that there is only one bus or one type of bus.

[0188] In one possible implementation, the processor 1110 is primarily responsible for interpreting computer program instructions (or codes) and processing data in the computer software. The computer program instructions and the data in the computer software may be stored in the memory 1120 or the cache of the processor 1110.

[0189] Optionally, processor 1110 may be an integrated circuit chip with signal processing capabilities. By way of example and not limitation, processor 1110 is a general-purpose processor, a digital signal processor (DSP), an ASIC, an FPGA, or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. A general-purpose processor is a microprocessor, for example. For example, processor 1110 is a central processing unit (CPU).

[0190] Memory 1120 can provide runtime space for processes in computing device 1100. For example, memory 1120 stores computer programs (specifically, program code) used to generate processes. After the processor executes the computer program to generate a process, the processor allocates corresponding storage space for the process in memory 1120. Furthermore, the aforementioned storage space further includes a text segment, an initialized data segment, a bit-initialized data segment, a stack segment, a heap segment, and the like. Memory 1120 stores data generated during the execution of the process, such as intermediate data or process data, in the storage space corresponding to the aforementioned process.

[0191] Alternatively, the memory is also referred to as the internal memory. It is used to temporarily store data being processed by the processor 1110, as well as data exchanged with external storage devices such as a hard disk. As long as the computer is running, the processor 1110 will load the data to be processed into the internal memory for processing, and then transmit the result after the calculation is completed.

[0192] By way of example and not limitation, memory 1120 is a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. The non-volatile storage medium may be, for example, a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory is a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus DRAM (DRDRAM). It should be noted that the memory 1120 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0193] The structure of the computing device 1100 listed above is only an example description, and the present application is not limited thereto. The computing device 1100 of the embodiment of the present application includes various hardware in the computer system in the prior art. For example, the computing device 1100 also includes other memories in addition to the memory 1120, such as disk storage, etc. Those skilled in the art should understand that the computing device 1100 may also include other devices necessary to achieve normal operation. At the same time, according to specific needs, those skilled in the art should understand that the above-mentioned computing device 1100 may also include hardware devices that implement other additional functions. In addition, those skilled in the art should understand that the above-mentioned computing device 1100 may also include only the devices necessary to implement the embodiment of the present application, and does not necessarily include all the devices shown in Figure 11.

[0194] The present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device may be a server. In some embodiments, the computing device may also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0195] As shown in Figure 12, the computing device cluster includes at least one computing device 1100. The memory 1120 of one or more computing devices 1100 in the computing device cluster may store the same instructions for executing the above method.

[0196] In some possible implementations, the memory 1120 of one or more computing devices 1100 in the computing device cluster may also store some instructions for executing the above method. In other words, the combination of one or more computing devices 1100 can jointly execute the instructions of the above method.

[0197] It should be noted that the memory 1120 in different computing devices 1100 in the computing device cluster can store different instructions, each for performing part of the functions of the above-mentioned apparatus. In other words, the instructions stored in the memory 1120 in different computing devices 1100 can implement the functions of one or more modules in the above-mentioned apparatus.

[0198] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. The network can be a wide area network (WAN) or a local area network (LAN), among others. FIG. 13 illustrates one possible implementation. As shown in FIG. 13 , two computing devices 1100A and 1100B are connected via a network. Specifically, each computing device is connected to the network via a communication interface within the computing device.

[0199] It should be understood that the functionality of the computing device 1100A shown in FIG13 may also be implemented by multiple computing devices 1100. Similarly, the functionality of the computing device 1100B may also be implemented by multiple computing devices 1100.

[0200] This embodiment also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on a computing device, it causes the computing device to perform the method provided above, or causes the computing device to implement the functions of the apparatus provided above.

[0201] This embodiment also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be executed on a computing device cluster or stored in any available medium. When executed by a computing device cluster, the computer program product causes the computing device cluster to perform the above-mentioned method or implement the functions of the above-mentioned apparatus.

[0202] This embodiment also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of storing data on a computing device, or a data storage device such as a data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, or a magnetic tape), an optical medium (e.g., a digital video disc (DVD)), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that, when executed on a computing device, cause the computing device to perform the method provided above.

[0203] This embodiment also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of storing data on a computing device, or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, or a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that, when executed by a computing device cluster, cause the computing device cluster to perform the method provided above.

[0204] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0205] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0206] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0207] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0208] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0209] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0210] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A data storage method, characterized in that: The method is applied to a cloud management platform, the cloud management platform is used to manage the infrastructure for providing cloud services, the infrastructure includes a storage device cluster, the storage device cluster includes a plurality of distributed storage devices, the method includes: Receive first request information from a tenant, where the first request information is used to obtain a first file; Determine, according to the first request information, a first data block set where the first file is located, the first data block set including one or more files, the files included in the first data block set have the same attributes, and the first data block set is stored in one of the multiple distributed storage devices; Determine, according to the first data block set, a first storage device where the first data block set is located; The first file is read from the first storage device.

2. The method according to claim 1, characterized in that The method further comprises: receiving second request information of the tenant, where the second request information is used to obtain multiple files, and the multiple files have the same attributes; Determine, according to the second request information, a second data block set where the plurality of files are located; Determine, according to the second data block set, a second storage device where the second data block set is located; The plurality of files are read from the second storage device.

3. The method according to claim 1 or 2, characterized in that: The attributes of the file include at least one of the following: identification information of the storage bucket to which the file belongs, the directory where the file is located, the storage time of the file, the storage type of the file, the size of the file, and the name of the file. The storage type of the file is used to indicate the access frequency of the file.

4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: receiving configuration information of the tenant, the configuration information comprising attributes of each file in at least one file; Determine N data block sets according to the configuration information and the at least one file, each data block set in the N data block sets includes one or more files in the at least one file, the files included in each data block set have the same attributes, the N data block sets include the first data block set, and N is a positive integer greater than or equal to 1; The N data block sets are stored in N storage devices in the at least one distributed storage device respectively.

5. The method according to claim 4, characterized in that The determining, according to the configuration information and the at least one file, N data block sets comprises: Determine at least one file set according to the attribute of each file in the at least one file, each file set including all files with the same attribute in the at least one file; According to the at least one file set, the N data block sets are determined, each data block set includes one or more files in a file set in the at least one file set, and the size of each data block set is less than or equal to a first preset threshold.

6. The method according to claim 4 or 5, characterized in that: The method further comprises: Determine M check data block sets according to the N data block sets, where the M check data block sets are used to perform recovery when one or more data block sets in the N data block sets fail, where M is a positive integer greater than or equal to 1; The M check data block sets are stored in M ​​storage devices in the at least one distributed storage device, where the M storage devices and the N storage devices are different storage devices.

7. The method according to claim 6, characterized in that Each data block set includes P data blocks, each check data block set includes P check data blocks, and determining M check data block sets according to the N data block sets includes: According to the p-th data block of each data block set in the N data block sets, determine the p-th check data block of each check data block set in the M check data block sets, the size of the p-th data block and / or the p-th check data block is less than or equal to a second preset threshold, p=1,...,P, P is a positive integer greater than or equal to 1.

8. The method according to claim 7, characterized in that The storing the N data block sets respectively in N storage devices in the at least one distributed storage device includes: storing the p-th data block of each data block set in the N data block sets in the N storage devices respectively; The storing the M check data block sets in M ​​storage devices in the at least one distributed storage device includes: The p-th check data block of each check data block set in the M check data block sets is stored in the M storage devices respectively.

9. The method according to claim 7 or 8, characterized in that: The P data blocks of a data block set are stored continuously in one storage device, and the P check data blocks of a check data block set are stored continuously in one storage device.

10. The method according to any one of claims 4 to 9, characterized in that The N data block sets have the same attributes, the attributes of each data block set are determined according to the attributes of the files included in each data block set, and the N storage devices are a group of storage devices.

11. The method according to any one of claims 4 to 10, characterized in that After determining N data block sets according to the configuration information and the at least one file, the method further includes: A first mapping relationship is generated, where the first mapping relationship is used to indicate identification information of a data block set where each file is located.

12. The method according to any one of claims 4 to 11, characterized in that After storing the N data block sets in N storage devices in the at least one distributed storage device respectively, the method further includes: A second mapping relationship is generated, where the second mapping relationship is used to indicate identification information of a storage device where each data block set is located.

13. A data storage device, characterized in that: The device is applied to a cloud management platform, the cloud management platform is used to manage the infrastructure for providing cloud services, the infrastructure includes a storage device cluster, the storage device cluster includes a plurality of distributed storage devices, and the device includes: A receiving module, configured to receive first request information from a tenant, where the first request information is used to obtain a first file; a processing module, configured to determine, according to the first request information, a first data block set where the first file is located, wherein the first data block set includes one or more files, the files included in the first data block set have the same attributes, and the first data block set is stored in one of the multiple distributed storage devices; The processing module is further used to determine the first storage device where the first data block set is located according to the first data block set; The processing module is further configured to read the first file from the first storage device.

14. The device according to claim 13, characterized in that The receiving module is further used to receive second request information of the tenant, where the second request information is used to obtain multiple files, and the multiple files have the same attributes; The processing module is further used to determine the second data block set where the multiple files are located according to the second request information; The processing module is further used to determine the second storage device where the second data block set is located according to the second data block set; The processing module is further configured to read the multiple files from the second storage device.

15. The device according to claim 13 or 14, characterized in that The attributes of the file include at least one of the following: identification information of the storage bucket to which the file belongs, the directory where the file is located, the storage time of the file, the storage type of the file, the size of the file, and the name of the file. The storage type of the file is used to indicate the access frequency of the file.

16. The device according to any one of claims 13 to 15, characterized in that The receiving module is further used to receive configuration information of the tenant, where the configuration information includes attributes of each file in at least one file; The processing module is further used to determine N data block sets according to the configuration information and the at least one file, each data block set in the N data block sets includes one or more files in the at least one file, the attributes of the files included in each data block set are the same, the N data block sets include the first data block set, and N is a positive integer greater than or equal to 1; The processing module is further used to store the N data block sets in N storage devices in the at least one distributed storage device respectively.

17. The device according to claim 16, characterized in that The processing module is specifically used for: Determine at least one file set according to the attribute of each file in the at least one file, each file set including all files with the same attribute in the at least one file; According to the at least one file set, the N data block sets are determined, each data block set includes one or more files in a file set in the at least one file set, and the size of each data block set is less than or equal to a first preset threshold.

18. The device according to claim 16 or 17, characterized in that The processing module is further used for: Determine M check data block sets according to the N data block sets, where the M check data block sets are used to perform recovery when one or more data block sets in the N data block sets fail, where M is a positive integer greater than or equal to 1; The M check data block sets are stored in M ​​storage devices in the at least one distributed storage device, where the M storage devices and the N storage devices are different storage devices.

19. The device according to claim 18, characterized in that Each data block set includes P data blocks, each check data block set includes P check data blocks, and the processing module is specifically used to: According to the p-th data block of each data block set in the N data block sets, the p-th check data block of each check data block set in the M check data block sets is determined, and the size of the p-th data block and / or the p-th check data block is less than or equal to The second preset threshold, p=1, ..., P, P is a positive integer greater than or equal to 1.

20. The device according to claim 19, characterized in that The processing module is specifically used for: storing the p-th data block of each data block set in the N data block sets in the N storage devices respectively; The p-th check data block of each check data block set in the M check data block sets is stored in the M storage devices respectively.

21. The device according to claim 19 or 20, characterized in that The P data blocks of a data block set are stored continuously in one storage device, and the P check data blocks of a check data block set are stored continuously in one storage device.

22. The device according to any one of claims 16 to 21, characterized in that The N data block sets have the same attributes, the attributes of each data block set are determined according to the attributes of the files included in each data block set, and the N storage devices are a group of storage devices.

23. The device according to any one of claims 16 to 22, characterized in that After determining N data block sets according to the configuration information and the at least one file, the processing module is further used to generate a first mapping relationship, where the first mapping relationship is used to indicate identification information of the data block set where each file is located.

24. The device according to any one of claims 16 to 23, characterized in that After storing the N data block sets in N storage devices in the at least one distributed storage device respectively, the processing module is further used to generate a second mapping relationship, where the second mapping relationship is used to indicate identification information of the storage device where each data block set is located.

25. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 12.

26. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster is caused to perform the method according to any one of claims 1 to 12.

27. A computer-readable storage medium, characterized in that: The method comprises computer program instructions, and when the computer program instructions are executed by a computing device cluster, the computing device cluster performs the method according to any one of claims 1 to 12.

Images