Data storage method and device, equipment and storage medium

Abstract

The embodiment of the invention discloses a data storage method and device, equipment and a storage medium, and relates to artificial intelligence and the Internet of Vehicles. The method comprises the steps that after to-be-stored data is obtained, a metadata label of the data is searched for, and the metadata label is used for representing business requirements of the data; in a plurality of data pools contained in the distributed file system, a data pool indicated by a data pool label matched with the metadata label is determined, and the data pool label is used for indicating the storage medium type of the corresponding data pool; and establishing a corresponding relationship between the data and the determined data pool, and sending a data identifier of the data and a data pool identifier corresponding to the data identifier to the distributed file system, so that the distributed file system stores the data corresponding to the data identifier into the data pool corresponding to the data pool identifier. By adopting the embodiment of the invention, the data storage efficiency can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data storage method, device, equipment and storage medium. Background Art

[0002] The distributed storage system (Ceph) can provide object, file, and block storage services at the same time. The distributed file system (Ceph File System, Cephfs) is a distributed file system service provided by Ceph, which can provide read and write services to multiple clients at the same time. Cephfs uses different data pools to store data, and different data pools can use different storage media. When using Cephfs, users often have different business needs. For example, some businesses that require high performance and low latency need to store data in a solid state disk (SSD) pool, while businesses that do not require much performance and latency need to store data in a large-capacity mechanical hard disk (HDD) pool.

[0003] Currently, the distinction between data pools in Cephfs requires confirmation by administrators and user businesses, followed by manual intervention. For example, if you want to adjust a newly created file from the HDD pool to the SSD pool with better performance, the administrator needs to manually switch the pool of the parent directory of the newly created file. This operation is cumbersome and results in low data storage efficiency. Summary of the invention

[0004] The embodiments of the present application provide a data storage method, apparatus, device, and storage medium, which do not require an administrator to manually switch pools of parent directories to which data to be stored belongs. The operation is convenient and can effectively improve data storage efficiency.

[0005] In a first aspect, an embodiment of the present application provides a data storage method, the method comprising:

[0006] After acquiring the data to be stored, searching for metadata tags of the data; wherein the metadata tags are used to characterize the business requirements of the data;

[0007] Determine, among a plurality of data pools included in the distributed file system, a data pool indicated by a data pool tag matching the metadata tag; wherein the data pool tag is used to indicate a storage medium type of the corresponding data pool;

[0008] Establishing a correspondence between the data and the determined data pool, and sending the data identifier of the data and the data pool identifier corresponding to the data identifier to the distributed file system, so that the distributed file system stores the data corresponding to the data identifier in the data pool corresponding to the data pool identifier. In one embodiment, the data storage method further includes:

[0009] Get directory metadata for any directory;

[0010] Adding a directory metadata extension attribute to the directory metadata; wherein the directory metadata extension attribute is used to indicate the storage medium type of the data pool associated with any of the directories;

[0011] The step of searching for the metadata tag of the data includes:

[0012] Determining directory metadata of the directory to which the data belongs;

[0013] The directory metadata extended attribute in the determined directory metadata is used as the metadata tag of the data. In one embodiment, the data storage method further includes:

[0014] Before writing data into any created file, obtaining file metadata of any file;

[0015] Adding a file metadata extension attribute to the file metadata; wherein the file metadata extension attribute is used to indicate the storage medium type of the data pool associated with any one of the files;

[0016] The step of searching for the metadata tag of the data includes:

[0017] determining file metadata for a file carrying the data;

[0018] The file metadata extension attribute in the determined file metadata is used as the metadata tag of the data. In one embodiment, the data storage method further includes:

[0019] After writing data in any created file, in response to a parameter configuration operation on any file, obtain the parameters configured for any file; wherein the parameters are used to indicate the storage medium type of the data pool associated with any file;

[0020] After receiving a request to send any of the files, converting the parameters into file metadata extended attributes; wherein the file metadata extended attributes include the parameters;

[0021] Adding the file metadata extended attribute to the file metadata of any file;

[0022] The step of searching for the metadata tag of the data includes:

[0023] determining file metadata for a file carrying the data;

[0024] In one embodiment, determining, among multiple data pools included in the distributed file system, a data pool indicated by a data pool tag that matches the metadata tag includes:

[0025] Obtaining data pool labels of each data pool included in the distributed file system; wherein the data pool label of any data pool is generated based on the storage medium type of the any data pool, and the data pool labels of data pools of different storage medium types are different;

[0026] Matching the data pool tags of each data pool with the metadata tag to determine the data pool indicated by the data pool tag matching the metadata tag. In one embodiment, determining the data pool indicated by the data pool tag matching the metadata tag among the multiple data pools included in the distributed file system includes:

[0027] Determine, among a plurality of data pools included in the distributed file system, at least one candidate data pool indicated by a data pool tag matching the metadata tag; wherein the storage medium type of each candidate data pool in the at least one candidate data pool is the same;

[0028] determining a target data pool from the at least one candidate data pool;

[0029] The establishing of the correspondence between the data and the determined data pool includes:

[0030] In one embodiment, determining the target data pool from the at least one candidate data pool includes:

[0031] Sorting the at least one candidate data pool to obtain a sorted candidate data pool;

[0032] Determine the data pool where data was last written;

[0033] The next data pool of the data pool determined in the sorted candidate data pools is used as the target data pool.

[0034] In a second aspect, an embodiment of the present application provides a data storage device, the data storage device comprising:

[0035] A search unit, configured to search for metadata tags of the data after acquiring the data to be stored; wherein the metadata tags are used to characterize the business requirements of the data;

[0036] A determination unit, configured to determine, among a plurality of data pools included in the distributed file system, a data pool indicated by a data pool tag matching the metadata tag; wherein the data pool tag is used to indicate a storage medium type of the corresponding data pool;

[0037] The sending unit is used to establish a corresponding relationship between the data and the determined data pool, and send the data identifier of the data and the data pool identifier corresponding to the data identifier to the distributed file system, so that the distributed file system stores the data corresponding to the data identifier in the data pool corresponding to the data pool identifier.

[0038] In a third aspect, an embodiment of the present application provides an electronic device, including a processor, a memory, and a transceiver, wherein the processor, the memory, and the transceiver are interconnected via a bus, the memory is used to store a computer program, the computer program includes program instructions, and the processor is configured to call the program instructions to perform the following steps:

[0039] After acquiring the data to be stored, searching for metadata tags of the data; wherein the metadata tags are used to characterize the business requirements of the data;

[0040] Determine, among a plurality of data pools included in the distributed file system, a data pool indicated by a data pool tag matching the metadata tag; wherein the data pool tag is used to indicate a storage medium type of the corresponding data pool;

[0041] A correspondence between the data and the determined data pool is established, and the data identifier of the data and the data pool identifier corresponding to the data identifier are sent to the distributed file system, so that the distributed file system stores the data corresponding to the data identifier in the data pool corresponding to the data pool identifier.

[0042] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the data storage method.

[0043] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program code. When the computer program code runs on a computer, the computer executes the method described in the first aspect above.

[0044] In a sixth aspect, an embodiment of the present application provides a computer program, which includes a computer program code. When the computer program code runs on a computer, the computer executes the method described in the first aspect above.

[0045] In the embodiment of the present application, by searching for the metadata tag of the data to be stored, and based on the metadata tag of the data, in the multiple data pools contained in the distributed file system, the data pool indicated by the data pool tag matching the metadata tag is determined, a corresponding relationship between the data and the determined data pool is established, and the data identifier of the data and the data pool identifier corresponding to the data identifier are sent to the distributed file system, so that the distributed file system stores the data in the determined data pool. It can be seen that the embodiment of the present application does not require the administrator to manually switch the pool operation of the parent directory to which the data to be stored belongs, so the embodiment of the present application is easy to operate and can effectively improve the storage efficiency of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background technology, the drawings required for use in the embodiments of the present application or the background technology will be described below.

[0047] Figure 1 It is a structural schematic diagram of a data storage system provided in an embodiment of the present application;

[0048] Figure 2 is a flow chart of a data storage method provided by an embodiment of the present application;

[0049] Figure 3 It is a flowchart of a data pool tag sending method provided in an embodiment of the present application;

[0050] Figure 4 is a schematic diagram of a matching scheduling method provided in an embodiment of the present application;

[0051] Figure 5 is a schematic diagram of the polling scheduling method provided in an embodiment of the present application;

[0052] Figure 6 It is a flowchart of another data storage method provided by an embodiment of the present application;

[0053] Figure 7 This is a schematic diagram of setting directory extension attributes provided by an embodiment of the present application;

[0054] Figure 8 It is a flowchart of another data storage method provided by an embodiment of the present application;

[0055] Fig. 9 This is a schematic diagram of setting file extension attributes provided by an embodiment of the present application;

[0056] Fig.10 It is a flowchart of another data storage method provided by an embodiment of the present application;

[0057] Fig.11 is a schematic diagram of a process for obtaining a data pool tag provided in an embodiment of the present application;

[0058] Fig.12 is a structural schematic diagram of a data storage device provided in an embodiment of the present application;

[0059] Fig.13 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] The embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application.

[0061] The embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc.

[0062] Cloud technology refers to a hosting technology that unifies hardware, software, network and other resources within a wide area network or local area network to achieve data computing, storage, processing and sharing.

[0063] Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool, which can be used on demand and is flexible and convenient. Cloud computing technology will become an important support. The backend services of the technical network system require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the rapid development and application of the Internet industry, in the future, each item may have its own identification mark, which needs to be transmitted to the backend system for logical processing. Data of different levels will be processed separately. All kinds of industry data need strong system backing support, which can only be achieved through cloud computing.

[0064] Among them, cloud storage is a new concept extended and developed from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of different types of storage devices (storage devices are also called storage nodes) in the network through application software or application interfaces to work together and provide external data storage and business access functions.

[0065] See also Figure 1 , Figure 11 is a schematic diagram of a data storage system provided by an embodiment of the present application. The data storage system may include a server and at least one client. The server may include Cephfs. Cephfs may include multiple data pools. Different data pools may use the same or different storage media. The storage media may include HDD or SSD, etc. Figure 1 For example, Cephfs may include two SSD pools (SSD pool 1 and SSD pool 2) and one HDD pool. Any client may include a directory tree, which may be used to indicate the structural relationship of each directory. Figure 1 For example, based on the directory tree in the client, it can be known that the client includes directory 1 and directory 2, wherein directory 1 includes file 1 and file 2, and directory 2 includes file 3, that is, the directory to which file 1 and file 2 belong is directory 1, and the directory to which file 3 belongs is directory 2.

[0066] The server can label each data pool based on the storage medium type of each data pool in Cephfs to obtain the data pool label of each data pool. The data pool labels of data pools with different storage medium types are different. For example, the data pool label of a data pool with SSD as the storage medium is "SSD", and the data pool label of a data pool with HDD as the storage medium is "HDD". After generating the data pool labels of each data pool, the server can send the data pool labels of each data pool to each client. It can be seen that each client can know the storage medium type of any data pool based on the data pool label of the data pool.

[0067] Any client can tag any data based on its business requirements. For example, if the business requirements of a certain data are high performance and low latency, then the data should be stored in the SSD pool, so the client can configure the metadata tag of the data as "SSD". For another example, if the business requirements of a certain data are not sensitive to performance and latency, then the data should be stored in the HDD pool, so the client can configure the metadata tag of the data as "HDD".

[0068] Based on the above description, after obtaining the data to be stored, any client can search for the metadata tag of the data, and based on the metadata tag of the data, determine the data pool indicated by the data pool tag that matches the metadata tag of the data in the multiple data pools contained in Cephfs. The client can establish a correspondence between the data and the determined data pool, and send the data identifier of the data and the data pool identifier corresponding to the data identifier to Cephfs. Cephfs can store the data in the data pool indicated by the data pool identifier corresponding to the data identifier of the data based on the data identifier of the data and the data pool identifier corresponding to the data identifier. Figure 1For example, since the metadata tag of file 1 is "SSD", the data in file 1 under directory 1 can be stored in SSD pool 1. Since the metadata tag of file 2 is "HDD", the data in file 2 under directory 1 can be stored in HDD pool. Based on this, through the data storage method provided in the embodiment of the present application, data with different business needs can be stored in different data pools, thereby realizing data hierarchical management.

[0069] It can be seen that in the embodiment of the present application, after the client obtains the data to be stored, it can search for the metadata tag of the data, and based on the metadata tag of the data, determine the data pool indicated by the data pool tag that matches the metadata tag of the data in the multiple data pools contained in Cephfs. There is no need for the administrator to manually switch the pool of the parent directory to which the data to be stored belongs. The embodiment of the present application is easy to operate and can effectively improve the storage efficiency of data.

[0070] Optionally, the client can store data of different business needs in the corresponding data pool in the server through multi-granularity tag scheduling. Among them, multi-granularity tag scheduling means that the client sets metadata tags of data of different business needs at two granularities, directory and file, and then stores data of different business needs in different data pools in the server through matching scheduling and polling scheduling.

[0071] Among them, the above-mentioned directory granularity is coarse-grained, which can be understood as, if the business requirements of the data contained in each file under the same directory are the same, then the data label can be generated in a coarse-grained manner, that is, the label of the directory is set, then the metadata label of the data contained in each file under the directory is the label of the directory, that is, the data contained in each file under the directory will be stored in the data pool indicated by the same data pool label.

[0072] The above file granularity is fine-grained, which can be understood as if the business requirements for the data contained in each file under the same directory are different, then the data labels can be generated in a fine-grained manner, that is, the labels of each file under the directory are set separately, then the data contained in different files under the directory will be stored in the data pool indicated by the data pool label that matches the metadata label of the data.

[0073] It can be seen that the multi-granularity is to meet the various needs of users and improve the configuration efficiency of metadata tags. For example, some users want to schedule all files under a certain directory to the data pool indicated by the same data pool tag, so they can set the tag of the directory. Some users only want to schedule a certain file under a certain directory to the data pool indicated by a certain data pool tag, so they can set the tag of the file.

[0074] Matching scheduling means that the client determines at least one candidate data pool indicated by a data pool tag that matches the metadata tag of the data to be stored in multiple data pools contained in Cephfs; polling scheduling means that after the client determines at least one candidate data pool indicated by a data pool tag that matches the metadata tag of the data, it determines the target data pool for storing the data from at least one candidate data pool. Figure 1 For example, since the metadata tag of file 1 is "SSD", the data pool tag that matches the metadata tag of file 1 can be determined to be "SSD" through matching scheduling. Figure 1 Cephfs includes two SSD pools (SSD pool 1 and SSD pool 2) and one HDD pool, so it can be determined that at least one candidate data pool includes SSD pool 1 and SSD pool 2. If the target data pool is determined to be SSD pool 1 from SSD pool 1 and SSD pool 2 through polling scheduling, the data in file 1 under directory 1 can be stored in SSD pool 1.

[0075] Among them, the server mentioned in the embodiments of the present application can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The client mentioned in the embodiments of the present application can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. The client and the server can be directly or indirectly connected via wired or wireless communication, and this application is not limited here.

[0076] The client in the embodiment of the present application may be run in an electronic device, including but not limited to a mobile phone, a computer, an intelligent voice interaction device, an intelligent home appliance, a vehicle terminal, an aircraft, etc.

[0077] See also Figure 2 , Figure 2 This is a flow chart of a data storage method provided by an embodiment of the present application, which method includes but is not limited to the following steps:

[0078] Step S201: The client obtains the data to be stored and searches for metadata tags of the data.

[0079] Optionally, the client may search for the directory to which the data belongs, determine the directory metadata of the directory, and use the directory metadata extended attribute in the determined directory metadata as the metadata tag of the data.

[0080] Optionally, the client may also search for the file to which the data belongs, determine the file metadata of the file, and use the file metadata extended attribute in the determined file metadata as the metadata tag of the data.

[0081] In one example, the client can first search for the directory metadata of the directory to which the data belongs. If the directory metadata contains a directory metadata extension attribute, it indicates that all data under the directory should be stored in the data pool indicated by the same data pool tag, that is, the data pool indicated by the data pool tag that matches the directory metadata extension attribute. Therefore, the directory metadata extension attribute is the metadata tag of the data, and the client can then execute steps S202 and S203.

[0082] If the directory metadata extended attribute does not exist in the directory metadata, it indicates that different data under the directory have different business requirements, that is, multiple data under the directory should be stored in data pools indicated by different data pool tags. For example, the business requirement of one data under the directory is high performance and low latency, so the data should be stored in the SSD pool. The business requirement of another data under the directory is not sensitive to performance and latency, so the other data should be stored in the HDD pool. Based on this, the client can search for the file metadata of the file to which the data belongs. The file metadata extended attribute in the file metadata is the metadata tag of the data, and then the client can execute steps S202 and S203. Step S202: The client determines the data pool indicated by the data pool tag that matches the metadata tag among the multiple data pools contained in the distributed file system.

[0083] Specifically, the client can obtain the data pool tags of each data pool, search for the data pool tag that matches the metadata tag from the obtained data pool tags, and then determine the data pool indicated by the data pool tag found above. Wherein, in the case where the naming method of the tag content of the data pool tag is the same as the naming method of the tag content of the metadata tag, the data pool tag that matches the metadata tag refers to: the data pool tag that is the same as the metadata tag. For example, assuming that the data pool tag of the data pool whose storage medium is SSD is "SSD", and the data pool tag of the data pool whose storage medium is HDD is "HDD". In addition, the metadata tag of the data to be stored is "SSD", then the client can determine that the data pool tag that is the same as the metadata tag "SSD" is "SSD", and then determine that the data pool indicated by the data pool tag "SSD" is the data pool used to store the data. In the case where the naming method of the tag content of the data pool tag is different from the naming method of the tag content of the metadata tag, the data pool tag that matches the metadata tag refers to: the data pool tag that has a mapping relationship with the metadata tag. For example, suppose the data pool label of a data pool whose storage medium is SSD is "SSD", and the data pool label of a data pool whose storage medium is HDD is "HDD". In addition, the metadata label of the data to be stored is "1". Since there is a mapping relationship between the metadata label "1" and the data pool label "HDD", the client can determine that the data pool label matching the metadata label "1" is "HDD", and further determine that the data pool indicated by the data pool label "HDD" is the data pool used to store the data.

[0084] Optionally, the server may label each data pool based on the storage medium type of each data pool in Cephfs to obtain a data pool label for each data pool. After generating the data pool label for each data pool, the server may send the data pool label for each data pool to each client. Therefore, after obtaining the data to be stored, the client may obtain the data pool label for each data pool locally on the client.

[0085] For example, see Figure 3 , Figure 3 The distributed file system in the server can obtain the storage medium type of any data pool in the multiple data pools it contains, and based on the storage medium type of any data pool, use the command "Ceph osd pool set <pool>medium <medium>" Generate a data pool tag for any data pool. The data pool tag for any data pool indicates its storage medium type.

[0086] The distributed file system sends the command to the master MON, triggering the master MON to modify and update the OSDMap and add the data pool tag to the attributes of the data pool.

[0087] The master MON can synchronize OSDMap to other slave MONs through the Paxos protocol. The client can synchronize OSDMap by subscribing and updating, so that the server can send the data pool labels of each data pool to the client. Among them, the Paxos protocol is a communication protocol used in distributed storage systems.

[0088] Among them, MON refers to the management service of Ceph, which is used to manage various topological information of multiple data pools contained in the distributed file system, such as OSDMap, MDSMap, etc.

[0089] Optionally, after the client obtains the data to be stored, the client locally obtains the data pool tags of each data pool. See the following embodiments for details. Fig.11 Related description.

[0090] Optionally, the client may determine the data pool indicated by the data pool tag matching the metadata tag among multiple data pools included in the distributed file system through multi-granularity tag scheduling, wherein the multi-granularity tag scheduling includes matching scheduling and polling scheduling.

[0091] First, the client determines at least one data pool indicated by a data pool tag that is the same as the metadata tag of the data among multiple data pools included in Cephfs through matching scheduling. Figure 4 As shown, the metadata tag of the data in file 2 under directory 1 is SSD, and the client can confirm at least one data pool indicated by the data pool tag that is the same as the metadata tag SSD, namely SSD pool 1 and SSD pool 2. The metadata tag of the data in file 3 under directory 2 is HDD, and the client can confirm at least one data pool indicated by the data pool tag that is the same as the metadata tag HDD, namely the HDD pool.

[0092] The client then determines the target data pool in the at least one data pool through polling scheduling. Specifically, the client sorts the at least one data pool to obtain a sorted data pool. The client determines the data pool where data was written last time, and uses the next data pool of the data pool determined in the sorted data pool as the target data pool.

[0093] like Figure 5 As shown, the metadata tag of the data to be stored is SSD, and the client can confirm at least one data pool SSD pool 1, SSD pool 2, SSD pool 3, and SSD pool 4 indicated by the data pool tag that is the same as the metadata tag SSD. SSD pool 1, SSD pool 2, SSD pool 3, and SSD pool 4 are recorded as SSD matching scheduling arrays, and the array subscripts of SSD pool 1, SSD pool 2, SSD pool 3, and SSD pool 4 are 0, 1, 2, and 3, respectively. In the figure, index is the SSD matching scheduling array subscript corresponding to the data pool that wrote data last time, and size is the total number of data pools in this figure, where size = 4. The client obtains the subscript of the target data pool through (index+1)%size, thereby determining the target data pool. For example, the client can determine that the data pool that wrote data last time is SSD pool 1, and the array subscript of SSD pool 1 is 0. Through (index+1)%size, it can be calculated that the array subscript of the target data pool is 1, and the target data pool is SSD pool 2.

[0094] Step S203: the client establishes a corresponding relationship between the data and the determined data pool, and sends the data identifier of the data and the data pool identifier corresponding to the data identifier to the distributed file system.

[0095] Optionally, after the client determines the target data pool from the at least one data pool, the client can establish a correspondence between the data and the target data pool, and send the data identifier of the data and the target data pool identifier corresponding to the data identifier to the distributed file system. The distributed file system can store the data corresponding to the data identifier in the target data pool corresponding to the target data pool identifier.

[0096] In an embodiment of the present application, after obtaining the data to be stored, the client can search for the metadata tag of the data, and based on the metadata tag of the data, determine the data pool indicated by the data pool tag that matches the metadata tag of the data in multiple data pools contained in Cephfs. There is no need for the administrator to manually switch the pool of the parent directory to which the data to be stored belongs. The embodiment of the present application is easy to operate and can effectively improve the storage efficiency of data.

[0097] See also Figure 6 , Figure 6 : is a flow chart of another data storage method provided in an embodiment of the present application, the method includes but is not limited to the following steps:

[0098] Step S601: The client obtains directory metadata of any directory.

[0099] Step S602: the client adds a new directory metadata extension attribute to the directory metadata, where the directory metadata extension attribute is used to indicate the storage medium type of the data pool associated with any directory.

[0100] like Figure 7 As shown, the client can obtain the directory metadata of directory 1, and add a new directory metadata extension attribute "ceph.dir.medium" to the directory metadata of directory 1. The newly added directory metadata extension attribute "ceph.dir.medium" includes an attribute value corresponding to the directory metadata extension attribute, and the attribute value is used to indicate the storage medium type of the data pool associated with the directory. Figure 7 The attribute value corresponding to the directory metadata extended attribute in is SSD. For example, the client can use "setfattr-n ceph.dir.medium-v <ssd hdd><Directory>" command to add directory metadata extended attributes to the directory metadata.

[0101] Among them, for a new directory, all files created under it will be scheduled to the data pool corresponding to the "ceph.dir.medium" extension attribute by default; for a directory that already contains files, only the newly created files under it will be affected, as shown in the diagram Figure 7 shown.

[0102] That is to say, if you add a directory metadata extension attribute "ceph.dir.medium" to the directory metadata of the created directory before writing data to the directory, then the metadata tags of all files created in the directory will refer to the attribute value corresponding to the directory metadata extension attribute, that is, the data of all files created in the directory will be stored in the data pool indicated by the data pool tag that matches the directory metadata extension attribute.

[0103] If a directory metadata extension attribute "ceph.dir.medium" is added to the directory metadata of the created directory after data is written to the directory, the metadata tag of the data written to the directory after the directory metadata extension attribute is added refers to the attribute value corresponding to the directory metadata extension attribute. The metadata tag of the data written to the directory before the directory metadata extension attribute is added refers to the file metadata extension attribute in the file metadata of the corresponding data.

[0104] Step S603: The client obtains the data to be stored and determines the directory metadata of the directory to which the data belongs.

[0105] Step S604: the client uses the directory metadata extension attribute in the determined directory metadata as the metadata tag of the data.

[0106] like Figure 7 As shown, assuming that the data to be stored is the data in file 2 under directory 1, since directory metadata of directory 1 has a newly added directory metadata extended attribute, the client can determine that the directory to which the data to be stored belongs is directory 1, determine the directory metadata of directory 1, and use the directory metadata extended attribute in the directory metadata of directory 1 as the metadata tag of the data. Among them, the directory metadata extended attribute in the directory metadata of directory 1 includes an attribute value corresponding to the directory metadata extended attribute.

[0107] Step S605: the client obtains a data pool label of each data pool included in the distributed file system, and the data pool label is generated based on the storage medium type of any data pool.

[0108] Optionally, the server may obtain the storage medium type of any data pool among the multiple data pools it contains, and generate a data pool tag of any data pool based on the storage medium type of the data pool, wherein the data pool tag of any data pool indicates its storage medium type.

[0109] Step S606: The client matches the data pool tags of each data pool with the metadata tag, and determines the data pool indicated by the data pool tag matching the metadata tag.

[0110] The client matches the data pool labels of each data pool with the metadata label through multi-granularity label scheduling, and determines the data pool indicated by the data pool label matching the metadata label. Figure 4 and Figure 5 shown.

[0111] Step S607: the client establishes a correspondence between the data and the determined data pool, and sends the data identifier of the data and the data pool identifier corresponding to the data identifier to the distributed file system.

[0112] After receiving the data identifier of the data sent by the client and the data pool identifier corresponding to the data identifier, the distributed file system may store the data in the data pool indicated by the data pool identifier corresponding to the data identifier of the data.

[0113] In this way, the client sets the metadata tag of the data by setting the directory extension attribute through steps S601 to S607, thereby storing the data in each file under a directory in a data pool of the same storage medium type without the need for manual operation by the administrator, thereby improving data storage efficiency.

[0114] See also Figure 8 , Figure 8 : is a flow chart of another data storage method provided in an embodiment of the present application, the method includes but is not limited to the following steps:

[0115] Step S801: before writing data in any created file, the client obtains the file metadata of any created file.

[0116] Step S802: the client adds a file metadata extension attribute to the file metadata, where the file metadata extension attribute is used to indicate the storage medium type of the data pool associated with any file.

[0117] like Fig. 9 As shown, the client can obtain the file metadata of file 1 under directory 1, and add a new file metadata extension attribute "ceph.file.medium" to the file metadata of file 1. The newly added file metadata extension attribute "ceph.file.medium" includes an attribute value corresponding to the file metadata extension attribute, and the attribute value is used to indicate the storage medium type of the data pool associated with the file. Fig. 9 The attribute value corresponding to the metadata extended attribute of the file is SSD.

[0118] Step S803: The client obtains the data to be stored and determines the file metadata of the file that carries the data.

[0119] Step S804: the client uses the file metadata extension attribute in the determined file metadata as the metadata tag of the data.

[0120] like Fig. 9 As shown, the data to be stored is the data in file 1 under directory 1, and the client can determine that the file used to carry the data to be stored is file 1, determine the file metadata of file 1, and use the file metadata extended attribute in the file metadata of file 1 as the metadata tag of the data. The file metadata extended attribute in the file metadata of file 1 includes the attribute value corresponding to the file metadata extended attribute.

[0121] Step S805: the client obtains a data pool label of each data pool included in the distributed file system. The data pool label is generated based on the storage medium type of any data pool.

[0122] Step S806: The client matches the data pool tags of each data pool with the metadata tag, and determines the data pool indicated by the data pool tag matching the metadata tag.

[0123] Optionally, the client can match the data pool labels of each data pool with the metadata label through multi-granularity label scheduling to determine the data pool indicated by the data pool label matching the metadata label. Figure 4 and Figure 5 shown.

[0124] Step S807: the client establishes a correspondence between the data and the determined data pool, and sends the data identifier of the data and the data pool identifier corresponding to the data identifier to the distributed file system.

[0125] After receiving the data identifier of the data sent by the client and the data pool identifier corresponding to the data identifier, the distributed file system may store the data in the data pool indicated by the data pool identifier corresponding to the data identifier of the data.

[0126] In this way, the client sets the metadata tag of the data by setting the file extension attribute through steps S801 to S807, thereby storing the data in any file in the data pool indicated by the data pool tag that matches the metadata tag of the data without the need for manual operation by the administrator, thereby improving data storage efficiency.

[0127] like Fig.10 As shown, in other embodiments, the client can set the file metadata extended attribute by setting a flag tag:

[0128] Step S1001: after writing data in any created file, the client obtains parameters configured for the file in response to a parameter configuration operation on the file, the parameters being used to indicate a storage medium type of a data pool associated with the file.

[0129] The above parameter is O_FLAG_MEDIUM, including O_MEDIUM_SSD parameter and O_MEDIUM_HDD parameter.

[0130] Step S1002: After receiving a request to send any file, the client converts the parameter into a file metadata extension attribute, where the file metadata extension attribute includes the parameter.

[0131] like Fig. 9 As shown, the client can convert the parameter O_FLAG_MEDIUM configured for any file in step S801 into a file metadata extension attribute through the computer command "open(pathname, O_RDWR|O_MEDIUM_SSD, mode)" or the computer command "open(pathname, O_RDWR|O_MEDIUM_HDD, mode)" after receiving a request to send any file. If the configured parameter O_FLAG_MEDIUM is O_MEDIUM_SSD, the parameter is converted into a file metadata extension attribute "ceph.dir.medium=SSD"; if the configured parameter O_FLAG_MEDIUM is O_MEDIUM_HDD, the parameter is converted into a file metadata extension attribute "ceph.dir.medium=HDD". Pathname is a file path character string of any file.

[0132] Step S1003: The client obtains the data to be stored and determines the file metadata of the file used to carry the data.

[0133] Step S1004: the client uses the file metadata extension attribute in the determined file metadata as the metadata tag of the data.

[0134] like Fig. 9 As shown, the data to be stored is the data in file 1 under directory 1, and the client can determine that the file used to carry the data to be stored is file 1, determine the file metadata of file 1, and use the file metadata extended attribute in the file metadata of file 1 as the metadata tag of the data. The file metadata extended attribute in the file metadata of file 1 includes the attribute value corresponding to the file metadata extended attribute.

[0135] Step S1005: the client obtains a data pool label of each data pool included in the distributed file system, and the data pool label is generated based on the storage medium type of any data pool.

[0136] For details about how distributed file systems generate data pool labels, see Figure 2 The embodiment shown.

[0137] Step S1006: the client matches the data pool tags of each data pool with the metadata tag, and determines the data pool indicated by the data pool tag matching the metadata tag.

[0138] The client matches the data pool labels of each data pool with the metadata label through multi-granularity label scheduling, and determines the data pool indicated by the data pool label matching the metadata label. Figure 4 and Figure 5 shown.

[0139] Step S1007: the client establishes a correspondence between the data and the determined data pool, and sends the data identifier of the data and the data pool identifier corresponding to the data identifier to the distributed file system.

[0140] After receiving the data identifier of the data sent by the client and the data pool identifier corresponding to the data identifier, the distributed file system may store the data in the data pool indicated by the data pool identifier corresponding to the data identifier of the data.

[0141] In this way, the client sets the file extension attributes by setting the flag tag through steps S1001 to S1007, thereby setting the metadata tag of the data, thereby storing the data in any file in the data pool indicated by the data pool tag that matches the metadata tag of the data, without the need for manual operation by the administrator, thereby improving data storage efficiency.

[0142] In the above embodiment, the client can obtain the data pool label of each data pool included in the distributed file system. The method for obtaining the data pool label can be referred to in Fig.11 As shown, the method of obtaining the data pool tag includes but is not limited to the following steps:

[0143] Step S1101: The client obtains file system details from fsmap according to the file system name.

[0144] The file system details include information such as data pool collection.

[0145] Step S1102: The client obtains a data pool set from the file system details.

[0146] Step S1103: The client obtains label attributes from the OSDMap according to each pool information.

[0147] Step S1104: The client aggregates the tag attribute set.

[0148] Specifically, the client can use "Ceph fs get<fs_name> The command "medium" obtains the data pool labels contained in the distributed file system. The command first obtains the set of data pools in the distributed file system through fsmap, and then obtains the data pool label information in each data pool through OSDMap.

[0149] The method of the embodiment of the present application is described in detail above, and the device of the embodiment of the present application is provided below.

[0150] See also Fig.12 , Fig.12 It is a structural schematic diagram of a data storage device provided in an embodiment of the present application. The data storage device may include a searching unit 1201, a determining unit 1202 and a sending unit 1203, wherein each unit is described in detail as follows.

[0151] The search unit 1201 is used to search for metadata tags of the data after acquiring the data to be stored; wherein the metadata tags are used to characterize the business requirements of the data;

[0152] The determining unit 1202 is used to determine, among multiple data pools included in the distributed file system, a data pool indicated by a data pool tag matching the metadata tag; wherein the data pool tag is used to indicate a storage medium type of the corresponding data pool;

[0153] The sending unit 1203 is used to establish a correspondence between the data and the determined data pool, and send the data identifier of the data and the data pool identifier corresponding to the data identifier to the distributed file system, so that the distributed file system stores the data corresponding to the data identifier in the data pool corresponding to the data pool identifier.

[0154] In one implementation, the data storage device may further include an acquisition unit 1204 and a new adding unit 1205, wherein:

[0155] The acquisition unit 1204 is used to acquire directory metadata of any directory;

[0156] A new adding unit 1205 is used to add a directory metadata extension attribute to the directory metadata; wherein the directory metadata extension attribute is used to indicate the storage medium type of the data pool associated with any directory;

[0157] The searching unit 1201 searches for the metadata tag of the data, including:

[0158] Determining directory metadata of the directory to which the data belongs;

[0159] The directory metadata extended attribute in the determined directory metadata is used as the metadata tag of the data.

[0160] In one implementation, the data storage device may further include:

[0161] The acquisition unit 1204 is used to acquire file metadata of any created file before writing data into the file;

[0162] A adding unit 1205 is used to add a file metadata extension attribute to the file metadata; wherein the file metadata extension attribute is used to indicate the storage medium type of the data pool associated with any one of the files;

[0163] The searching unit 1201 searches for the metadata tag of the data, including:

[0164] determining file metadata for a file carrying the data;

[0165] The file metadata extended attribute in the determined file metadata is used as the metadata tag of the data.

[0166] In one implementation, the data storage device may further include:

[0167] The acquisition unit 1204 is used to, after writing data in any created file, respond to a parameter configuration operation on any file to acquire the parameters configured on the any file; wherein the parameters are used to indicate the storage medium type of the data pool associated with the any file;

[0168] The conversion unit 1206 is used to convert the parameter into a file metadata extended attribute after receiving the sending request for the any file; wherein the file metadata extended attribute includes the parameter;

[0169] A adding unit 1205, configured to add the file metadata extended attribute to the file metadata of any file;

[0170] The searching unit 1201 searches for the metadata tag of the data, including:

[0171] determining file metadata for a file carrying the data;

[0172] The file metadata extended attribute in the determined file metadata is used as the metadata tag of the data.

[0173] In one implementation, the determining unit 1202 determines, among multiple data pools included in the distributed file system, a data pool indicated by a data pool tag matching the metadata tag, including:

[0174] Obtaining data pool labels of each data pool included in the distributed file system; wherein the data pool label of any data pool is generated based on the storage medium type of the any data pool, and the data pool labels of data pools of different storage medium types are different;

[0175] The data pool tags of the respective data pools are matched with the metadata tags to determine the data pool indicated by the data pool tags matching the metadata tags.

[0176] In one implementation, the determining unit 1202 determines, among multiple data pools included in the distributed file system, a data pool indicated by a data pool tag matching the metadata tag, including:

[0177] Determine, among a plurality of data pools included in the distributed file system, at least one candidate data pool indicated by a data pool tag matching the metadata tag; wherein the storage medium type of each candidate data pool in the at least one candidate data pool is the same;

[0178] determining a target data pool from the at least one candidate data pool;

[0179] The sending unit 1203 establishes a correspondence between the data and the determined data pool, including:

[0180] A corresponding relationship between the data and the target data pool is established.

[0181] In one implementation, the determining unit 1202 determines the target data pool from the at least one candidate data pool, including:

[0182] Sorting the at least one candidate data pool to obtain a sorted candidate data pool;

[0183] Determine the data pool where data was last written;

[0184] The next data pool of the data pool determined in the sorted candidate data pools is used as the target data pool.

[0185] See also Fig.13 , Fig.13 It is a structural diagram of an electronic device 1300 provided in an embodiment of the present application. The electronic device 1300 includes a processor 1301, a memory 1302 and a transceiver 1303. The processor 1301, the memory 1302 and the transceiver 1303 are interconnected via a bus.

[0186] The memory 1302 includes, but is not limited to, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), or a compact disc read-only memory (CD-ROM), and is used for related instructions and data. The transceiver is used to receive and send data.

[0187] The processor 1301 may be one or more central processing units (CPUs). When the processor 1301 is a CPU, the CPU may be a single-core CPU or a multi-core CPU.

[0188] The processor 1301 in the device 1300 is used to read the program code stored in the memory 1302 and perform the following operations:

[0189] After acquiring the data to be stored, searching for metadata tags of the data; wherein the metadata tags are used to characterize the business requirements of the data;

[0190] Determine, among a plurality of data pools included in the distributed file system, a data pool indicated by a data pool tag matching the metadata tag; wherein the data pool tag is used to indicate a storage medium type of the corresponding data pool;

[0191] Establishing a correspondence between the data and a determined data pool;

[0192] The transceiver 1303 in the electronic device performs the following operations:

[0193] The data identifier of the data and the data pool identifier corresponding to the data identifier are sent to the distributed file system, so that the distributed file system stores the data corresponding to the data identifier in the data pool corresponding to the data pool identifier.

[0194] In one implementation, the processor 1301 in the electronic device is further configured to perform the following operations:

[0195] Get directory metadata for any directory;

[0196] Adding a directory metadata extension attribute to the directory metadata; wherein the directory metadata extension attribute is used to indicate the storage medium type of the data pool associated with any of the directories;

[0197] When searching for the metadata tag of the data, the processor 1301 is used to perform the following operations:

[0198] Determining directory metadata of the directory to which the data belongs;

[0199] The directory metadata extended attribute in the determined directory metadata is used as the metadata tag of the data.

[0200] In one implementation, the processor 1301 in the electronic device is further configured to perform the following operations:

[0201] Before writing data into any created file, obtaining file metadata of any file;

[0202] Adding a file metadata extension attribute to the file metadata; wherein the file metadata extension attribute is used to indicate the storage medium type of the data pool associated with any one of the files;

[0203] When searching for the metadata tag of the data, the processor 1301 is used to perform the following operations:

[0204] determining file metadata for a file carrying the data;

[0205] The file metadata extended attribute in the determined file metadata is used as the metadata tag of the data.

[0206] In one implementation, the processor 1301 in the electronic device is further configured to perform the following operations:

[0207] After writing data in any created file, in response to a parameter configuration operation on any file, obtain the parameters configured for any file; wherein the parameters are used to indicate the storage medium type of the data pool associated with any file;

[0208] After receiving a request to send any of the files, converting the parameters into file metadata extended attributes; wherein the file metadata extended attributes include the parameters;

[0209] Adding the file metadata extended attribute to the file metadata of any file;

[0210] When searching for the metadata tag of the data, the processor 1301 is used to perform the following operations:

[0211] determining file metadata for a file carrying the data;

[0212] The file metadata extended attribute in the determined file metadata is used as the metadata tag of the data.

[0213] In one implementation, the processor 1301 in the electronic device is configured to perform the following operations when determining, among multiple data pools included in the distributed file system, a data pool indicated by a data pool tag matching the metadata tag:

[0214] Obtaining data pool labels of each data pool included in the distributed file system; wherein the data pool label of any data pool is generated based on the storage medium type of the any data pool, and the data pool labels of data pools of different storage medium types are different;

[0215] The data pool tags of the respective data pools are matched with the metadata tags to determine the data pool indicated by the data pool tags matching the metadata tags.

[0216] In one implementation, the processor 1301 in the electronic device is configured to perform the following operations when determining, among multiple data pools included in the distributed file system, a data pool indicated by a data pool tag matching the metadata tag:

[0217] Determine, among a plurality of data pools included in the distributed file system, at least one candidate data pool indicated by a data pool tag matching the metadata tag; wherein the storage medium type of each candidate data pool in the at least one candidate data pool is the same;

[0218] determining a target data pool from the at least one candidate data pool;

[0219] When establishing the correspondence between the data and the determined data pool, the processor 1301 is used to perform the following operations:

[0220] A corresponding relationship between the data and the target data pool is established.

[0221] In one implementation, the processor 1301 in the electronic device is configured to perform the following operations when determining the target data pool from the at least one candidate data pool:

[0222] Sorting the at least one candidate data pool to obtain a sorted candidate data pool;

[0223] Determine the data pool where data was last written;

[0224] The next data pool of the data pool determined in the sorted candidate data pools is used as the target data pool.

[0225] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.< / ssd> < / medium> < / pool>

Claims

1. A data storage method, characterized in that: include: After acquiring the data to be stored, searching for metadata tags of the data; wherein the metadata tags are used to characterize the business requirements of the data; Determine, among a plurality of data pools included in the distributed file system, a data pool indicated by a data pool tag matching the metadata tag; wherein the data pool tag is used to indicate a storage medium type of the corresponding data pool; A correspondence between the data and the determined data pool is established, and the data identifier of the data and the data pool identifier corresponding to the data identifier are sent to the distributed file system, so that the distributed file system stores the data corresponding to the data identifier in the data pool corresponding to the data pool identifier.

2. The method according to claim 1, characterized in that The method further comprises: Get directory metadata for any directory; Adding a directory metadata extension attribute to the directory metadata; wherein the directory metadata extension attribute is used to indicate the storage medium type of the data pool associated with any of the directories; The step of searching for the metadata tag of the data includes: Determining directory metadata of the directory to which the data belongs; The directory metadata extended attribute in the determined directory metadata is used as the metadata tag of the data.

3. The method according to claim 1, characterized in that The method further comprises: Before writing data into any created file, obtaining file metadata of any file; Adding a file metadata extension attribute to the file metadata; wherein the file metadata extension attribute is used to indicate the storage medium type of the data pool associated with any one of the files; The step of searching for the metadata tag of the data includes: determining file metadata for a file carrying the data; The file metadata extended attribute in the determined file metadata is used as the metadata tag of the data.

4. The method according to claim 1, characterized in that The method further comprises: After writing data in any created file, in response to a parameter configuration operation on any file, obtain the parameters configured for any file; wherein the parameters are used to indicate the storage medium type of the data pool associated with any file; After receiving a request to send any of the files, converting the parameters into file metadata extended attributes; wherein the file metadata extended attributes include the parameters; Adding the file metadata extended attribute to the file metadata of any file; The step of searching for the metadata tag of the data includes: determining file metadata for a file carrying the data; The file metadata extended attribute in the determined file metadata is used as the metadata tag of the data.

5. The method according to claim 1, characterized in that The step of determining, among a plurality of data pools included in the distributed file system, a data pool indicated by a data pool tag matching the metadata tag comprises: Obtaining data pool labels of each data pool included in the distributed file system; wherein the data pool label of any data pool is generated based on the storage medium type of the any data pool, and the data pool labels of data pools of different storage medium types are different; The data pool tags of the respective data pools are matched with the metadata tags to determine the data pool indicated by the data pool tags matching the metadata tags.

6. The method according to claim 1, characterized in that The step of determining, among a plurality of data pools included in the distributed file system, a data pool indicated by a data pool tag matching the metadata tag comprises: Determine, among a plurality of data pools included in the distributed file system, at least one candidate data pool indicated by a data pool tag matching the metadata tag; wherein the storage medium type of each candidate data pool in the at least one candidate data pool is the same; determining a target data pool from the at least one candidate data pool; The establishing of the correspondence between the data and the determined data pool includes: A corresponding relationship between the data and the target data pool is established.

7. The method according to claim 6, characterized in that The determining of the target data pool from the at least one candidate data pool comprises: Sorting the at least one candidate data pool to obtain a sorted candidate data pool; Determine the data pool where data was last written; The next data pool of the data pool determined in the sorted candidate data pools is used as the target data pool.

8. A data storage device, characterized in that: The device comprises: A search unit, configured to search for metadata tags of the data after acquiring the data to be stored; wherein the metadata tags are used to characterize the business requirements of the data; A determination unit, configured to determine, among a plurality of data pools included in the distributed file system, a data pool indicated by a data pool tag matching the metadata tag; wherein the data pool tag is used to indicate a storage medium type of the corresponding data pool; The sending unit is used to establish a corresponding relationship between the data and the determined data pool, and send the data identifier of the data and the data pool identifier corresponding to the data identifier to the distributed file system, so that the distributed file system stores the data corresponding to the data identifier in the data pool corresponding to the data pool identifier.

9. An electronic device, characterized in that: The method comprises a processor, a memory and a transceiver, wherein the processor, the memory and the transceiver are interconnected via a bus, the memory is used to store a computer program, the computer program comprises program instructions, and the processor is configured to call the program instructions to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 7.

Images