Data service method and apparatus

Abstract

A data service method and device are disclosed. Based on a file system data organization mode, metadata of to-be-stored data is generated. The metadata is used to describe an object name of the to-be-stored data in an object storage service. The object storage service is called to store the to-be-stored data. Thus, the file system and the object storage can be fused into a unified data storage mechanism from the bottom. While having the file system capability and the object storage capability, the cost of maintaining two sets of data storage mechanisms can be reduced.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of data, in particular to a data service method and device. BACKGROUND

[0002] A data lake refers to a system that stores data in natural formats such as large binary objects or files. It usually stores all enterprise data uniformly, including both original copies in source systems and converted data such as data for reporting, visualization, data analysis and machine learning. A data lake can include structured data of relational databases (rows and columns), semi-structured data (CSV, logs, XML, JSON), unstructured data (emails, files, PDFs) and binary data (images, audio, video).

[0003] In a data lake scenario, applications want to use the features of object storage, such as terabyte-level file number scalability, Tbps-level bandwidth capability, and cost-effective advantages of data life cycle. However, many applications in the data lake scenario are built based on HDFS in the early stage, which provides file system-like functions such as directory atomic operations, snapshots, file append writing (append) and truncation (truncate). Early applications are programmed in the same way as using a POSIX file system.

[0004] A file system uses a directory hierarchy, which has technical challenges in supporting the scalability of a large number of files. Object storage uses a flat namespace, which makes it difficult to support directory atomic operations, snapshots, file append writing (append) and truncation (truncate) and other file system functions. During the evolution based on the data lake, enterprises have both (old) application environments built based on HDFS clusters and (new) application environments rebuilt based on object storage, resulting in the problems of operation and maintenance costs of two sets of systems and the cost competitiveness of HDFS cluster technology.

[0005] Therefore, there is a need for a solution that can reduce operation and maintenance costs while having both file system capabilities and object storage capabilities. SUMMARY

[0006] One of the technical problems to be solved by the present disclosure is to provide a solution that can reduce operation and maintenance costs while having both file system capabilities and object storage capabilities.

[0007] According to a first aspect of the present disclosure, a data service method is provided, comprising: generating metadata of to-be-stored data based on a data organization manner of a file system, the metadata being used to describe an object name of the to-be-stored data in an object storage service; and calling the object storage service to store the to-be-stored data.

[0008] Optionally, the metadata comprises address mapping information of each of the at least one data block obtained by splitting the to-be-stored data, and the address mapping information comprises a first object name of the data block in the object storage service; and the step of invoking the object storage service to store the to-be-stored data comprises: invoking the object storage service to store the data block as the first object.

[0009] Optionally, the method further comprises: in response to triggering of a preset condition, invoking the object storage service to store the to-be-stored data as a second object; and recording a second object name of the second object in the object storage service in the metadata.

[0010] Optionally, the first object name is used to represent an object storage bucket where the data block is located and a prefix of the data block, and / or the second object name is used to represent an absolute path of the to-be-stored data.

[0011] Optionally, the method further comprises: configuring a first domain name and a second domain name, the first domain name being an access domain name of a file service system used to provide data services for users, and the second domain name being a domain name of an object storage bucket associated with the file service system.

[0012] According to a second aspect of the present disclosure, a data service method is provided, comprising: in response to a file service system creation request, generating file service system metadata; invoking an object storage service to set an object storage bucket associated with the file service system; transmitting at least part of the file service system metadata into the object storage bucket to generate object storage service metadata in the object storage bucket, the object storage service metadata being capable of representing an association relationship between the file service system and the object storage bucket; and updating the file service system metadata, so that the updated file service system metadata is capable of representing the association relationship between the file service system and the object storage bucket.

[0013] Optionally, the method further comprises: configuring a first domain name and a second domain name, the first domain name being an access domain name of the file service system, and the second domain name being a domain name of an object storage bucket associated with the file service system.

[0014] Optionally, the method further comprises: in response to a data write request based on the first domain name, splitting the to-be-stored data, and generating metadata of each of the at least one data block obtained by the splitting, the metadata comprising address mapping information of the data block, and the address mapping information comprising a first object name of the data block in the object storage service; and invoking the object storage service to store the data block as the first object.

[0015] Optionally, the method further comprises: in response to triggering of a preset condition, invoking the object storage service to store the to-be-stored data as a second object, and recording a second object name of the second object in the object storage service in the metadata.

[0016] Optionally, the first object name is used to represent an object storage bucket where the data block is located and a prefix of the data block, and / or the second object name is used to represent an absolute path of the data to be stored.

[0017] Optionally, the method further comprises: in response to the data write request based on the second domain name, finding the first domain name associated with the second domain name; and converting the data write request based on the second domain name into a data write request based on the first domain name.

[0018] Optionally, the method further comprises: in response to the data read request based on the first domain name, obtaining metadata of the data to be read; and invoking the object storage service to read the data according to the metadata of the data to be read.

[0019] Optionally, the method further comprises: in response to the data read request based on the second domain name, shielding a prefix used to represent a data block in metadata of the data to be read; and invoking the object storage service to read the data based on the shielded metadata.

[0020] Optionally, the method further comprises: if the object storage service cannot read the data based on the shielded metadata, converting the data read request based on the second domain name into a data read request based on the first domain name.

[0021] According to a third aspect of the present disclosure, a data service apparatus is provided, comprising: a generation module configured to generate metadata of data to be stored based on a data organization manner of a file system, the metadata being used to describe an object name of the data to be stored in an object storage service; and an invocation module configured to invoke the object storage service to store the data to be stored.

[0022] According to a fourth aspect of the present disclosure, a data service apparatus is provided, comprising: a generation module configured to generate file service system metadata in response to a file service system creation request; a setting module configured to invoke an object storage service to set an object storage bucket associated with the file service system; a transmission module configured to transmit at least part of the file service system metadata into the object storage bucket to generate object storage service metadata in the object storage bucket, the object storage service metadata being capable of representing an association relationship between the file service system and the object storage bucket; and an update module configured to update the file service system metadata so that the updated file service system metadata is capable of representing the association relationship between the file service system and the object storage bucket.

[0023] According to a fifth aspect of the present disclosure, a computing device is provided, comprising: a processor; and a memory having executable code stored thereon, the executable code, when executed by the processor, causing the processor to perform the method according to the first aspect or the second aspect.

[0024] According to a sixth aspect of the present disclosure, a non-transitory machine-readable storage medium is provided, having stored thereon executable code that, when executed by a processor of an electronic device, causes the processor to perform the method according to the first aspect or the second aspect.

[0025] Thus, the present disclosure generates metadata of to-be-stored data based on a file system-based data organization manner, and stores the to-be-stored data by calling an object storage service, so that the file system and the object storage can be fused into a unified data storage mechanism from the bottom layer, and the cost of maintaining two sets of data storage mechanisms can be reduced while the file system capability and the object storage capability are both possessed. BRIEF DESCRIPTION OF DRAWINGS

[0026] The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which like reference characters refer to the like parts throughout the figures, and in which:

[0027] Figure 1 A schematic flowchart of a data service method according to an embodiment of the present disclosure is shown.

[0028] Figure 2 A storage system architecture of the present disclosure that fuses a file system and an object storage is shown.

[0029] Figure 3 A data organization structure and a hot and cold tiering structure of the storage system of the present disclosure are shown.

[0030] Figure 4 A creation flowchart of a file service system is shown.

[0031] Figure 5 A data writing flowchart based on different domain names is shown.

[0032] Figure 6 A data reading flowchart based on different domain names is shown.

[0033] Figure 7 A structure schematic diagram of a data service apparatus according to an embodiment of the present disclosure is shown.

[0034] Figure 8 A structure schematic diagram of a data service apparatus according to another embodiment of the present disclosure is shown.

[0035] Figure 9 A structure schematic diagram of a computing device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0036] The preferred embodiments of the present disclosure will be described in more detail with reference to the drawings. Although the preferred embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0037] File system and object storage are two different data storage mechanisms.

[0038] File system stores files in a directory hierarchy, and can provide file system functions such as directory atomic operation, snapshot, file append and truncate. The file system can refer to a distributed file system.

[0039] Taking HDFS as an example, HDFS refers to a distributed file system designed to run on commodity hardware. HDFS was originally developed as the infrastructure of the Apache Nutch search engine project, which is part of the Apache Hadoop Core project. HDFS has many commonalities with existing distributed file systems, but it also has obvious differences from other distributed file systems. HDFS is a highly fault-tolerant system suitable for deployment on inexpensive machines. HDFS can provide high-throughput data access and is very suitable for large-scale data sets. HDFS relaxes some POSIX constraints to achieve the purpose of streaming file system data.

[0040] Object storage, also known as object-based storage, is a general term used to describe a method of solving and processing discrete units, which are referred to as objects. Like files, objects contain data, but unlike files, objects do not have a hierarchical structure in a layer structure. Each object is at the same level in a flat address space called a storage pool, and an object does not belong to the next level of another object.

[0041] File system uses directory hierarchy, and there are technical challenges in supporting the scalability of large-scale file numbers; while object storage uses flat name space, it is difficult to support file system functions such as directory atomic operation, snapshot, file append and truncate.

[0042] In view of this, the present disclosure proposes to fuse the two data storage mechanisms of file system and object storage into a unified data storage mechanism from the bottom-up, so as to reduce the cost of maintaining two data storage mechanisms.

[0043] Figure 1 A schematic flowchart of a data service method according to one embodiment of the present disclosure is shown.

[0044] Referring to Figure 1 In step S110, metadata of the to-be-stored data is generated based on the file system-based data organization manner, and the metadata is used to describe an object name of the to-be-stored data in the object storage service.

[0045] The metadata generated based on the file system-based data organization manner refers to that the metadata information of the to-be-stored data, such as the directory tree of the to-be-stored data, can be determined according to the file system-based data organization manner, so that the generated metadata can reflect the data organization characteristics of the file system, that is, the directory tree structure. That is, the metadata can be organized in the directory tree manner.

[0046] In step S120, the object storage service is called to store the to-be-stored data.

[0047] That is, the to-be-stored data can actually be stored as an object in an object storage bucket provided by the object storage service.

[0048] The bucket is a container for storing objects in the object storage, and all objects belong to a certain storage space (that is, the bucket). The object storage provides a flat storage manner based on the bucket and the object, and all objects in the bucket are at the same logical level.

[0049] Therefore, the present disclosure can integrate the file system and the object storage into a unified data storage mechanism from the bottom layer by means of "generating metadata based on the file system-based data organization manner + object storage of data", and can reduce the cost of maintaining two sets of data storage mechanisms while having the file system capability and the object storage capability.

[0050] The unified data storage mechanism of the present disclosure can provide both the file system capability and the object storage capability. When storing data based on the data storage mechanism of the present disclosure, a corresponding storage manner can be selected according to the storage requirement or data characteristics of the to-be-stored data, so that the stored data has a corresponding capability.

[0051] Example 1

[0052] If the to-be-stored data needs to be frequently changed, such as performing append, truncate and other operations on the to-be-stored data later, when storing the to-be-stored data, the stored data needs to have the file system capability to facilitate the execution of the append, truncate and other operations on the data. The data storage process is as follows.

[0053] In the step S110, the metadata can be organized in a directory tree manner, and a tree structure storage capability of root directory, subdirectory, file, etc. can be provided, and the directory atomization operation can be supported by the tree structure organization.

[0054] In the process of organizing the metadata, the to-be-stored data can be split based on the sharding storage feature of the file system to obtain at least one data block, and metadata including address mapping information of each of the at least one data block is generated, and the address mapping information includes an object name (for the convenience of distinction, it can be referred to as a first object name) of the data block in the object storage service.

[0055] In the process of reading the data, the metadata is read first, and then the data is read from the object storage service according to the metadata.

[0056] In order to enable the data block to be successfully read from the object storage service according to the first object name, the first object name can be named according to a naming rule that enables the first object name to represent the object storage bucket where the data block is located and the prefix of the data block.

[0057] As an example, the data block can be saved under / bucket-name / fs-prefix / prefix, bucket-name is used to represent the bucket name associated with the data block (or the file system where the data block is located), and fs-prefix is used to represent the prefix of the data block mapping obtained by splitting the to-be-stored data.

[0058] In the step S120, the data block can be stored as a first object by calling the object storage service.

[0059] That is, each data block can be stored as an object in the corresponding object storage bucket.

[0060] The data storage mechanism of the present disclosure "generating metadata based on a file system data organization manner + object storage of data" can provide file system capability by organizing the metadata in a data slicing manner and storing each data block obtained by splitting by calling the object storage service.

[0061] Example 2

[0062] If the to-be-stored data is read-light and write-light, emphasizes the expansibility of the number of files, and the cost performance, when storing the to-be-stored data, the stored data needs to have object storage capability. The data storage process is as follows.

[0063] In the step S110, the metadata can be organized in a directory tree manner, and a tree structure storage capability of root directory, subdirectory, file, etc. can be provided, and the directory atomization operation can be supported by the tree structure organization.

[0064] The generated metadata can record the absolute path of the to-be-stored data, and the absolute path is taken as an object name (for the sake of distinction, it can be referred to as a second object name) of the to-be-stored data in the object storage service.

[0065] The absolute path refers to a path from a root directory at the top of a tree directory structure to a certain directory or file (in the present disclosure, to a file where the to-be-stored data is located). The second object name can also be referred to as an absolute path name.

[0066] When step S120 is performed, the object storage service can be invoked to store the to-be-stored data as a second object.

[0067] That is, the entire to-be-stored data is stored as one object. By directly storing the to-be-stored data as an object for object storage, the scalability of object storage can be achieved. Therefore, the data storage mechanism of the present disclosure “generating metadata based on a file system data organization method + storing data as an object” can provide object storage capability.

[0068] The present disclosure also supports data migration. Data migration refers to that, after data is stored based on the file system capability or the object storage capability provided by the present disclosure, the data can be switched from storage based on one capability (file system capability or object storage capability) to storage based on another capability (object storage capability or file system capability).

[0069] Specifically, after the data is stored according to example 1, the data can also be converted to be stored according to example 2, so that the data is switched from having file system capability to having object storage capability.

[0070] Correspondingly, after the data is stored according to example 2, the data can also be converted to be stored according to example 1, so that the data is switched from having object storage capability to having file system capability.

[0071] The following is an exemplary application scenario of the data migration function.

[0072] Most of the new data written are hot data, and there are more computing access modes for hot data, such as file append write and truncation, temporary computing directory renaming, and snapshot protection.

[0073] Hot data will become cold data after a period of time, and cold data almost does not need to be changed, that is, it is read more and updated less, and cold data emphasizes the scalability and cost-effectiveness of the number of files.

[0074] Through the above analysis, it can be concluded that hot data needs file system capability, and cold data needs object storage capability.

[0075] Therefore, for newly-arriving data to be stored (i.e. hot data), the method shown in Example 1 can be used for storage, so that the hot data can have file system capabilities.

[0076] In response to triggering of a preset condition (e.g. hot data becoming cold data after being stored for more than a predetermined length of time, or a user request), the hot data can be converted to cold data storage according to the method shown in Example 2, so that the cold data can have object storage capabilities.

[0077] When the hot data is stored according to the method shown in Example 2, the address mapping information in the metadata can be deleted after the hot data is successfully converted to cold data for object storage. Wherein, storing the data to be stored as a second object (i.e. converting the hot data to cold data for object storage) by calling the object storage service can mean that the object storage service combines the data blocks obtained by previously storing the data to be stored into one object for storage. That is, the hot data can be converted to cold data only by changing the metadata.

[0078] The present disclosure can configure two access domain names, which are the access domain name of the file service system for providing data services for users (i.e. the first domain name) and the domain name of the object storage bucket associated with the file service system (i.e. the second domain name). Users can access the file service system through the first domain name to obtain data read and write services, and can also access the file service system through the first domain name to obtain data read and write services. The data read and write processes based on different domain names are described in detail below.

[0079] Figure 2 The storage system architecture of the present disclosure is shown, which integrates file system and object storage. Figure 2 The file system is taken as HDFS for example, and it should be understood that other types of file systems can also be used.

[0080] As shown in Figure 2 The storage system can be set in a computing server located in the cloud. The storage system provides HDFS interface access and object storage interface access. The storage system mainly consists of HDFS SDK, "HDFS metadata" and object storage service.

[0081] The HDFS SDK is a file storage client. The HDFS SDK can be embedded in a big data analysis platform, and a typical implementation is Hadoop SDK, through which HDFS interface access and object storage interface access can be achieved.

[0082] When the storage system provides file read and write services externally, the "HDFS metadata" provides metadata capabilities. The "HDFS metadata" refers to a server or functional module capable of providing metadata capabilities. The metadata managed by the "HDFS metadata" can include metadata generated for the to-be-stored data when the data is stored, and file service system metadata when providing file read and write services externally. The file service system refers to a system created for a user to provide file read and write services for the user.

[0083] When the HDFS SDK (such as Hadoop SDK) accesses the file service system created for the user, the "HDFS metadata" is read first to obtain metadata, and then the "object storage service" is accessed to read data.

[0084] The object storage service is the existing object storage technology, which provides object data access for the client and provides storage space for the file read and write service, that is, the data of the file read and write service (such as HDFS service) is saved in the object storage.

[0085] When creating a file service system (such as HDFS file system), the object storage bucket (Bucket) supported by the backend needs to be specified, at this time, two domain names of the Endpoint (terminal address) for accessing the file service system can be obtained.

[0086] The HDFS domain name corresponds to the first domain name mentioned above. The HDFS domain name is the recommended domain name, and the application can access the data based on the domain name. The HDFS domain name can be an AZ-level domain name or a Region-level domain name, and the AZ-level domain name is recommended, for example, hdfs-name.az.oss-cn-shenzhen.XXX.com.

[0087] The bucket domain name corresponds to the second domain name mentioned above. The bucket domain name is the object storage bucket domain name associated with the file service system, and it is not recommended for the application to use it directly, but when data distribution through the object storage is needed, other applications can directly access the data through the Internet. The domain name maintains the existing mechanism of the object storage, for example, datalake-bucket.oss-cn-shenzhen.XXX.com.

[0088] The two domain names can be configured on the HDFS SDK, and the application uses the HDFS domain name by default to read and write data.

[0089] Figure 3 The data organization structure and cold and hot hierarchical structure of the storage system of the present disclosure are shown.

[0090] The metadata intercommunication between the "HDFS metadata" and the object storage service shown in the figure can include the intercommunication between the file service system metadata and the object storage bucket information, as well as the mutual processing between the running processes related to the intercommunication metadata.

[0091] When the storage system provides external file storage services, "HDFS metadata" can organize metadata in a directory tree format, providing storage capabilities for root directories (such as "root" in the figure), subdirectories (such as "subdirectory 1, subdirectory 2, subdirectory 3"), and files (such as "hot files, cold files" in the figure).

[0092] Hot files can record the mapping information of each address block obtained by segmenting the data. Cold files can record stubs that describe the absolute path of the cold file in the file service system.

[0093] When a newly written file is a hot file, the hot file will be sliced ​​into a certain size, so the metadata of the file content is composed of multiple "address block mapping groups". For example, if a 100MB file is sliced ​​into 1MB, then the metadata of the first 1MB is the address block 1 mapping. The address block 1 mapping stores the object name of the 1MB data on the object storage. The object naming format is md5(fileid)-offset-size. The related data structure can be defined as

[0094] struct{

[0095] int64 block-id;

[0096] string obj-name; / / Object naming rules, such as / bucket-name / fs-prefix / md5(fileid)-offset-size

[0097] }

[0098] Therefore, the data blocks of a specific file service system are stored under the prefix / bucket-name / fs-prefix / , where bucket-name is the bucket name associated with the user file system, and fs-prefix is ​​the prefix for storing content slices of the user file service system.

[0099] After the HDFS SDK obtains the file metadata information from the "HDFS metadata", it can obtain the object name of the data content to be read through the address block mapping table, so that it can directly read the data from the object storage.

[0100] Hot data is organized in a tree structure to support directory atomic operations, and metadata is organized in a data slice manner to conveniently support file append and truncation operations.

[0101] Hot data becomes cold data after a period of time, and the application will not modify (e.g., will not perform directory renaming, and will not modify the file) for the cold data. At this time, the hot data can be transferred to the object storage, for example, the 100MB file described above can be directly stored as an independent object. When the hot data is successfully converted into a cold data object, the hot data will delete the metadata of the "data block mapping", and use the metadata to record a "stub", which records the object name on the object storage. The object name corresponds to the absolute path of the file in the HDFS file system.

[0102] For example, the hot data file / test / 2021-5 / datalake-file in HDFS is converted into a cold data file, and the object name on the object storage remains the same, that is, / test / 2021-5 / datalake-file, and the "stub" of the HDFS metadata corresponding to the file metadata is changed to the object name. In this way, the hot and cold layering can directly read the cold data file on the object storage, and because the HDFS metadata is simplified to a "stub" record, the size of the HDFS metadata is reduced, and the scalability of the object storage is fully utilized to manage cold data, and finally the scalability of the data lake fusion storage is realized.

[0103] Before using the data service of the present disclosure, the user needs to create a file service system first to obtain an Endpoint (which can be represented by an HTTP address) for accessing the file service system. The user can access the Endpoint to read and write data. The Endpoint can be configured with two access domain names, that is, the first domain name and the second domain name mentioned above.

[0104] The creation of the file service system and the data read and write after the creation of the file service system are described below.

[0105] Figure 4 The creation process of the file service system is shown in the schematic diagram.

[0106] In step S410, in response to the file service system creation request, the file service system metadata is generated.

[0107] Before providing file services to the user, the user needs to create a file service system. The user can request to create a file service system by logging into the related application. Figure 4 The console in the application can refer to the server corresponding to the application to which the user logs in.

[0108] The "HDFS metadata" records the user information, so that the file service system created for the user can be found later.

[0109] The file service system metadata can refer to information related to the created file service system, such as (key) configuration data of the file service system, typically including the file service system name (for example: test-spark), description information (for example: XXXX test file service system), and permission information (for example, private access < only the user can read and write >).

[0110] The user ID information provided by the user can be carried in the file service system creation request. The "HDFS metadata" can also record this user ID information when generating the file service system metadata.

[0111] In step S420, an object storage bucket is created, and at least part of the file service system metadata is transmitted into the object storage bucket.

[0112] The object storage service can be called to set (create) the object storage bucket associated with the file service system, which is used as a container for data in the file service system.

[0113] The file service system metadata transmitted into the object storage bucket can refer to the key configuration data of the file service system, such as the file service system name, description information, and permission information, so as to minimize invalid parameters.

[0114] In step S430, object storage service metadata is generated.

[0115] The object storage service metadata can record the association (i.e. mapping relationship) between the user, the file service system, and the object storage bucket, to help subsequent reverse lookup.

[0116] For example, the "user, file service system metadata, and object storage bucket" can be associated as a database record, which is an extension of the object storage metadata, so the record saves the mapping relationship. The user and the file service system metadata are the contents of step S410, and the object storage bucket is the content introduced according to the object storage technology principle.

[0117] In step S440, the file service system metadata is updated.

[0118] The updated file service system metadata can associate the object storage bucket with the file service system. The user information can also be recorded when generating the file service system metadata, so that the updated file service system metadata can reflect the association between the user, the file service system, and the object storage bucket.

[0119] When creating the file service system, two domain names for accessing the file service system can be configured, i.e. the first domain name and the second domain name mentioned above. After the file service system is created successfully, the first domain name and the second domain name can be returned.

[0120] The user can access the file service system through the first domain name to obtain data read-write services, and can also obtain data read-write services through the second domain name. The first domain name can be a recommended domain name. The second domain name is not recommended to be directly used, but when data distribution needs to be performed through object storage, other applications can directly access data through the Internet.

[0121] 1. Data writing process based on different domain names

[0122] In response to a data writing request based on the first domain name, the data can be stored according to the description of example 1 above.

[0123] In response to a data writing request based on the second domain name, the data writing request based on the second domain name can be redirected to a data writing request based on the first domain name to store the data according to the description of example 1 above. For example, the first domain name associated with the second domain name can be found, and the data writing request based on the second domain name can be converted to a data writing request based on the first domain name, and redirected and forwarded to the first domain name.

[0124] The following takes the first domain name as the HDFS domain name and the second domain name as the bucket domain name as an example, and combines the above description. Figure 5 The data writing process based on the HSFS domain name and the data writing process based on the bucket domain name are further described respectively.

[0125] (a) Data writing process based on HSFS domain name

[0126] Step 1: The file can be created by HDFS SDK, and data can be written.

[0127] Step 2: The metadata and data can be prepared by “HDFS metadata”.

[0128] Step 3: The data is sliced according to a specified size (such as 1 MB), each slice is named according to a specific rule, and each slice is written as an object into an object storage bucket provided by an object storage service. The naming rule can be referred to the description above.

[0129] Step 4: The object storage service can write the object according to the normal object storage process.

[0130] Step 5: The writing success is returned.

[0131] Step 6: The metadata and data are submitted.

[0132] Step 7: The success is returned.

[0133] (b) Data writing process based on bucket domain name

[0134] Step 1: The file can be created by HDFS SDK, and data can be written.

[0135] Step 2, the HDFS domain name associated with the bucket domain name can be checked by the object storage service.

[0136] Step 3, return the redirection address of the HDFS domain name.

[0137] The request is forwarded to the HDFS domain name through redirection, and then the write process of the DFS domain name is reused. Through this redirection forwarding mechanism, the consistency of the write requests of the two domain names is realized.

[0138] 2. Data read process based on different domain names

[0139] In response to a data read request based on the first domain name, the metadata of the data to be read can be obtained first, and then the object storage service is called to read the data according to the metadata of the data to be read.

[0140] In response to a data read request based on the second domain name, the prefix (such as / fs-prefix) used to represent the data block in the metadata of the data to be read needs to be shielded first, because it is an object name used internally by the file service system, so it can only be read through the HDFS SDK; then the object storage service is called to read the data based on the shielded metadata; if the object storage service cannot be called to read the data based on the shielded metadata, the data read request based on the second domain name is redirected and converted into a data read request based on the first domain name.

[0141] The following takes the first domain name as the HDFS domain name and the second domain name as the bucket domain name as an example, and combines the above Figure 6 The data read process based on the HSFS domain name and the data read process based on the bucket domain name are further described respectively.

[0142] (a) Data read process based on HSFS domain name

[0143] Step 1, the HDFS SDK can send a request to read file metadata to the "HDFS metadata".

[0144] Step 2, in response to the metadata read request, the "HDFS metadata" can read the metadata. The read metadata can be address block mapping or stub.

[0145] Step 3, the "HDFS metadata" can return a read success message to the HDFS SDK, and the read metadata is carried in the message.

[0146] Step 4, call the object storage service object storage according to the metadata (address block mapping or stub) to read file data.

[0147] Step 5, read the object according to the metadata by the object storage service.

[0148] Step 6, return success.

[0149] The whole process is around the metadata organization format of HDFS, and data is read from object storage through the obtained address block mapping or stub record.

[0150] (b) Data reading process based on bucket domain name

[0151] Step 1, the HDFS SDK can send a request to read file metadata to the object storage service

[0152] Step 2, the object storage service masks the address block mapping prefix ( / fs-prefix), directly reads the object, and returns the content if successful, or returns a redirection address of the HDFS domain name. The request is forwarded to the HDFS domain name through redirection, and the reading process of the DFS domain name is reused.

[0153] Step 3, return success.

[0154] The core of the whole process is in step 2, which first masks the address block mapping prefix request reading, because this is an object name used internally by HDFS, so it can only be read through the HDFS SDK, and the reading of other APPs will return an exception.

[0155] If the object name of the data reading is not in the address block mapping prefix (for example, / fs-prefix), it is accessed as a normal object. If the object exists (for example, the data is hot and the whole file is saved to the object storage by the hot and cold classification function of HDFS, and only the "stub" in the HDFS metadata), the object data is directly accessed. If the object does not exist, the data may be hot data, at which time the request is forwarded to the HDFS domain name through redirection, and then the reading process of the HDFS domain name is reused.

[0156] Through this multiple judgment mechanism, the consistency of the reading requests of the two domain names is realized.

[0157] In the process of evolution based on data lake, enterprises have both (old) application environments based on HDFS clusters and (new) application environments based on object storage reconstruction, resulting in the problems of operation and maintenance costs of two systems and the lack of cost competitiveness of HDFS cluster technology.

[0158] The data service scheme of the present disclosure which integrates the file system capability and the object storage capability supports the directory atomic operation, snapshot, file append write and truncation function of the old application to HDFS, and meets the large-scale expansion capability of the new application to massive data storage, and therefore can be applied to data lake to realize unified data lake, provide truly cloud-native data lake storage for big data computing and analysis application, and provide high cost performance.

[0159] The data service method of the present disclosure can also be implemented as a data service device for providing data service for users. The data service device can be set in the cloud.

[0160] Figure 7 A structural diagram of the data service device according to one embodiment of the present disclosure is shown. The functional units of the data service device can be implemented by hardware, software or a combination of hardware and software that implements the principles of the present disclosure. Those skilled in the art can understand that, Figure 7 The described functional units can be combined or divided into sub-units to implement the principles of the above-mentioned invention. Therefore, the description herein can support any possible combination, or division, or further limitation of the functional units described herein.

[0161] The functional units that the data service device can have and the operations that each functional unit can perform are briefly described below. For the details involved, please refer to the relevant description above, which will not be repeated here.

[0162] Referring to Figure 7 , the data service device 700 can include a generation module 710 and a calling module 720.

[0163] The generation module 710 is configured to generate metadata of the to-be-stored data based on a data organization manner of a file system, the metadata being used to describe an object name of the to-be-stored data in an object storage service. The calling module 720 is configured to call the object storage service to store the to-be-stored data.

[0164] For the specific process of the data service device 700 storing data through the generation module 710 and the calling module 720, please refer to the relevant description above, such as the description above in combination with Example 1, Example 2 and other relevant descriptions.

[0165] The data service device 700 can also include a configuration module configured to configure a first domain name and a second domain name, the first domain name being an access domain name of a file service system for providing data service for users, and the second domain name being a domain name of an object storage bucket associated with the file service system.

[0166] Figure 8 A structural diagram of the data service device according to another embodiment of the present disclosure is shown.

[0167] Referring to Figure 8 , the data service device 800 can include a generation module 810, a setting module 820, a transmission module 830 and an updating module 840.

[0168] The generation module 810 is configured to generate file service system metadata in response to a file service system creation request. The setting module 820 is configured to call the object storage service to set an object storage bucket associated with the file service system. The transfer module 830 is configured to transfer at least a portion of the file service system metadata to the object storage bucket to generate object storage service metadata in the object storage bucket that can represent the association relationship between the file service system and the object storage bucket. The update module 840 is configured to update the file service system metadata so that the updated file service system metadata can represent the association relationship between the file service system and the object storage bucket.

[0169] The data service device 800 may further include a configuration module for configuring a first domain name and a second domain name, wherein the first domain name is an access domain name of a file service system for providing data services to users, and the second domain name is a domain name of an object storage bucket associated with the file service system.

[0170] The data service device 800 may further include a segmentation module, a generation module, and a call module. In response to a data write request based on the first domain name, the segmentation module segments the data to be stored, and the generation module generates metadata for at least one data block obtained by the segmentation, the metadata including address mapping information of the data block, and the address mapping information including the first object name of the data block in the object storage service; the call module calls the object storage service to store the data block as the first object

[0171] In response to the triggering of the preset condition, the calling module can call the object storage service to store the data to be stored as the second object, and record the second object name of the second object in the object storage service in the metadata.

[0172] The data service device 800 may further include a search module and a conversion module. In response to a data write request based on the second domain name, the search module searches for the first domain name associated with the second domain name. The conversion module converts the data write request based on the second domain name into a data write request based on the first domain name.

[0173] The data service device 800 may further include an acquisition module and a first reading module. In response to a data read request based on the first domain name, the acquisition module acquires metadata of the data to be read. The reading module calls the object storage service to read the data based on the metadata of the data to be read.

[0174] The data service device 800 may further include a masking module and a second reading module. In response to a data read request based on the second domain name, the masking module masks a prefix representing a data block in metadata of the requested data; and the second reading module calls the object storage service to read the data based on the masked metadata.

[0175] The data service apparatus 800 can further include a conversion module configured to convert the data reading request based on the second domain name into the data reading request based on the first domain name if the object storage service cannot read the data based on the shielded metadata.

[0176] Figure 9 A structural diagram of a computing device according to an embodiment of the present disclosure is shown, which can be used to implement the data service method described above.

[0177] Referring to Figure 9 The computing device 900 includes a memory 910 and a processor 920.

[0178] The processor 920 can be a multi-core processor or can include multiple processors. In some embodiments, the processor 920 can include a general-purpose main processor and one or more special-purpose coprocessors, such as a graphics processor (GPU), a digital signal processor (DSP), and the like. In some embodiments, the processor 920 can be implemented using a customized circuit, such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0179] The memory 910 can include various types of storage units, such as a system memory, a read-only memory (ROM), and a permanent storage device. Among them, the ROM can store static data or instructions required by the processor 920 or other modules of the computer. The permanent storage device can be a read-write storage device. The permanent storage device can be a non-volatile storage device that does not lose stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device uses a mass storage device (such as a magnetic or optical disk, a flash memory) as a permanent storage device. In some other embodiments, the permanent storage device can be a removable storage device (such as a floppy disk, an optical drive). The system memory can be a read-write storage device or a volatile read-write storage device, such as a dynamic random access memory. The system memory can store some or all of the instructions and data required by the processor during runtime. In addition, the memory 910 can include a combination of any computer readable storage media, including various types of semiconductor storage chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), magnetic disks and / or optical disks. In some embodiments, the memory 910 can include a read and / or write removable storage device, such as a compact disc (CD), a read-only digital versatile disc (such as DVD-ROM, double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (such as an SD card, a min SD card, a Micro-SD card, etc.), a magnetic floppy disk, etc. The computer readable storage medium does not include a carrier wave and a transient electronic signal transmitted through wireless or wired transmission.

[0180] The memory 910 stores executable code, which, when processed by the processor 920, can cause the processor 920 to perform the data service method described above.

[0181] The data service method, apparatus and device according to the present disclosure have been described in detail above with reference to the accompanying drawings.

[0182] In addition, the method according to the present disclosure can also be implemented as a computer program or a computer program product, which includes computer program code instructions for performing each step defined in the above method of the present disclosure.

[0183] Alternatively, the present disclosure can also be implemented as a non-transitory machine readable storage medium (or computer readable storage medium, or machine readable storage medium) having executable code (or computer program, or computer instruction code) stored thereon, which, when executed by a processor of an electronic device (or computing device, server, etc.), causes the processor to perform each step of the above method according to the present disclosure.

[0184] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein can be implemented as electronic hardware, computer software, or combinations of both.

[0185] The flow diagrams and the block diagrams in the drawings are presented to illustrate the architecture, functionality, and operation of possible implementations of systems and methods in accordance with the present disclosure. In this regard, each block in the flow diagrams and the block diagrams can represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the block can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may

[0186] Embodiments of the present disclosure have been described above, and the description is intended to be illustrative of the embodiments and not restrictive. Many modifications and variations of the described embodiments are possible and are apparent to those of ordinary skill in the art. The selection of the terms to be used in the description is not intended to limit the scope of the embodiments described herein, but rather to best describe the principles of the embodiments in the best mode contemplated by the inventors. The scope of the embodiments is to be determined by the following claims.

Claims

1.A data service method, comprising: generating metadata of to-be-stored data based on a file system-based data organization manner, the metadata being used to describe an object name of the to-be-stored data in an object storage service; the metadata comprising address mapping information of each of at least one data block obtained by splitting the to-be-stored data, the address mapping information comprising a first object name of the data block in the object storage service, the first object name being used to represent an object storage bucket where the data block is located and a prefix of the data block; storing the data block as a first object by calling the object storage service; storing the to-be-stored data as a second object by calling the object storage service in response to triggering of a preset condition; recording a second object name of the second object in the object storage service in the metadata, the second object name being used to represent an absolute path of the to-be-stored data. 2.A data service method, comprising: generating file service system metadata in response to a file service system creation request; setting an object storage bucket associated with the file service system by calling an object storage service; transferring at least part of the file service system metadata into the object storage bucket to generate object storage service metadata in the object storage bucket, the object storage service metadata being capable of representing an association relationship between the file service system and the object storage bucket; updating the file service system metadata so that the updated file service system metadata is capable of representing the association relationship between the file service system and the object storage bucket; further comprising: the metadata comprising address mapping information of each of at least one data block obtained by splitting the to-be-stored data, the address mapping information comprising a first object name of the data block in the object storage service, the first object name being used to represent an object storage bucket where the data block is located and a prefix of the data block; storing the data block as a first object by calling the object storage service; storing the to-be-stored data as a second object by calling the object storage service in response to triggering of a preset condition; recording a second object name of the second object in the object storage service in the metadata, the second object name being used to represent an absolute path of the to-be-stored data. 3.The method of claim 2, further comprising: configuring a first domain name and a second domain name, the first domain name being an access domain name of the file service system, and the second domain name being a domain name of the object storage bucket associated with the file service system. 4.The method of claim 3, further comprising: in response to a data write request based on the first domain name, splitting the to-be-stored data and generating metadata of each of at least one data block obtained by the splitting. 5.The method of claim 4, further comprising: in response to a data write request based on the second domain name, finding a first domain name associated with the second domain name; converting the data write request based on the second domain name into a data write request based on the first domain name. 6.The method of claim 3, further comprising: in response to a data read request based on the first domain name, obtaining metadata of to-be-read data; According to metadata of the data to be read, the object storage service is invoked to read the data. 7.The method of claim 3, further comprising: in response to the data read request based on the second domain name, masking the prefix in the metadata of the data to be read for characterizing the data block; invoking the object storage service to read the data based on the masked metadata. 8.The method of claim 7, further comprising: if the object storage service fails to read the data based on the masked metadata, converting the data read request based on the second domain name to a data read request based on the first domain name. 9.A computing device comprising: a processor; and a memory having stored thereon executable code that, when executed by the processor, causes the processor to perform the method of any of claims 1-8. 10.A computer program product comprising executable code that, when executed by a processor of an electronic device, causes the processor to perform the method of any of claims 1-8. 11.A non-transitory machine-readable storage medium having stored thereon executable code that, when executed by a processor of an electronic device, causes the processor to perform the method of any of claims 1-8. ​

Images