Hybrid storage systems, methods, and media

Abstract

The application provides a hybrid storage system, method and medium, comprising a client, a data interaction module, a file system adapter and an underlying storage system; the client accesses user requests in real time, parses the user requests and extracts user operation request sentences; the data interaction module converts the operation request sentences into operation sentences of the corresponding underlying storage system through the file system adapter, performs operations and returns operation results to the client; and the client returns the operation results to the user. The application realizes free conversion of a file system tree structure to a key value mode of an object storage system; and unstructured data such as audio and video and structured data under the file system can be directly accessed without the need for business code adaptation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of hybrid storage, in particular to a hybrid storage system, method and medium. BACKGROUND

[0002] Hybrid storage is a solution for storing digital data that integrates flash storage, which provides excellent performance and fast access speed, with media types that provide higher cost efficiency. Hybrid storage enables organizations to store data on a single platform while balancing performance and economic benefits.

[0003] The Chinese invention patent document with publication number CN112084156A discloses a hybrid storage system. A file entry component receives a file to be backed up, and the file entry component generates metadata of the file, which records the activity level, file size, and online backup time of the file. The file distribution component has a rule controller that applies a distribution rule to the metadata to obtain a target storage, the distribution rule being related to the activity level, file size, and online backup time of the file, and the file distribution component sends the file to the target storage. The large file online storage cluster stores files with a file size greater than a size threshold, an activity level less than an activity threshold, and an online backup time that has not expired. The small file online storage cluster stores files with a file size not greater than the size threshold, an activity level less than the activity threshold, and an online backup time that has not expired. The hot data cache cluster stores files with an activity level not less than the activity threshold. The permanent backup library stores files with an online backup time that has expired.

[0004] According to the related technology in the above, the inventors believe that the data access interface compatibility is poor, the data structure has differences, which easily causes data islands, and the business end-to-end process is relatively complex. SUMMARY

[0005] In view of the defects in the prior art, the purpose of the present application is to provide a hybrid storage system, method and medium.

[0006] According to the hybrid storage system provided by the present application, the client, the data interaction module, the file system adapter and the underlying storage system are included.

[0007] The client accesses user requests in real time, parses user requests, and extracts user operation request statements.

[0008] The data interaction module converts the operation request statement into an operation statement corresponding to the underlying storage system through the file system adapter, and returns the operation result to the client.

[0009] The client returns the operation result to the user.

[0010] Preferably, the system further comprises a metadata management module;

[0011] The data interaction module obtains a file path name through an operation request statement;

[0012] The data interaction module accesses the metadata management module through the file path name to obtain the underlying storage system corresponding to the file path;

[0013] The data interaction module converts the operation request statement into an operation statement of the underlying storage system corresponding to the file path through the file system adapter, performs operation, and returns the operation result to the client.

[0014] Preferably, the client provides a unified data access service interface for users and processes user requests.

[0015] Preferably, the data interaction module obtains a data storage location through an operation request statement;

[0016] The underlying storage system comprises a file system and an object storage system;

[0017] The data interaction module is initialized based on a configuration file, accesses the file system or the storage system, and registers a unique identifier for each underlying storage system according to the data storage location.

[0018] Preferably, the data interaction module obtains a creation time through an operation request statement, retains information for a file, and queries the file creation time.

[0019] Preferably, the metadata management module manages storage system metadata;

[0020] The storage system metadata is related data of all accessed underlying storage systems, and subsequent operation requests are accessed to the underlying storage system for operation through the related data.

[0021] Preferably, the metadata management module manages data file metadata;

[0022] The data file metadata comprises structured data and unstructured data;

[0023] The file system adapter performs unified program interface of the structured data and the unstructured data.

[0024] Preferably, the file system adapter performs free conversion of file system and object storage system structure:

[0025] The object storage system performs a uniform program interface operation of a file system: limiting namespace operations under the same bucket space, taking the bucket as a root directory of the file system, and subsequently creating a file or a directory under the file system; taking an absolute path of the file or the directory as a key, and taking written file content as a value, to perform file system mode writing and reading; after conversion, the file system lists all files under a directory, and performs prefix query on the entire key table;

[0026] The file tree system performs a key value storage design mode: corresponding to the uniform program interface operation of the object storage system file system, taking the key in the object storage system as an absolute path of a file; creating a key value pair as storing value file content under the key path in the file system.

[0027] According to the method for mixed storage provided by the application, the system for mixed storage comprises the following steps:

[0028] The client step: accessing a user request in real time, and parsing the user request to extract a user operation request statement;

[0029] The processing step: converting the operation request statement into an operation statement of a corresponding bottom layer storage system to perform operation, and obtaining an operation result;

[0030] The returning step: returning the operation result to the user.

[0031] According to the medium for mixed storage provided by the application, the system for mixed storage is included.

[0032] Compared with the prior art, the application has the following beneficial effects:

[0033] 1. The application realizes free conversion of a file system tree structure to a key value mode of an object storage system;

[0034] 2. The application can directly access unstructured data such as audio and video under the object storage system and structured data under the file system without the need for business code adaptation;

[0035] 3. The application solves the unified access demand of upper layer businesses such as machine learning and data mining for structured data and unstructured data by adopting mapping of a file system tree structure to a key value structure, and simplifies a business implementation process. BRIEF DESCRIPTION OF DRAWINGS

[0036] Other characteristics, objects and advantages of the application will become more apparent through reading the following detailed description of the non-limiting embodiments with reference to the accompanying drawings:

[0037] Figure 1 The system architecture diagram of the application;

[0038] Figure 2 The file system adapter function diagram is shown in FIG. 1.

[0039] Figure 3 The Chinese representation diagram is shown in FIG. 2. Figure 2 DETAILED DESCRIPTION

[0040] The application will be described in detail below with specific embodiments. The following embodiments will help those skilled in the art to further understand the application, but do not limit the application in any form. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the application. These are within the scope of the application.

[0041] According to the hybrid storage system provided by the application, as shown in FIG. 1 and FIG. 2, the system comprises a client, a data interaction module, a metadata management module, a file system adapter and an underlying storage system. Figure 1 Figure 2 The client accesses the user request in real time, parses the user request, and extracts the user operation request statement. The client provides a unified data access service interface for the user and processes the user request.

[0042] Specifically, the client module provides a unified data access service interface for the user. The interface provides a unified port to receive user requests such as reading and writing data. The main function of the module is to process user requests, including communication of upper layer computing jobs or business application programs (representing possible application programs calling the interface). First, the user access request is processed, then the task is issued to the data interaction module, and finally the processing result is returned to the upper layer user.

[0043] Specifically, the client module provides a unified data access service interface for the user. The interface provides a unified port to receive user requests such as reading and writing data. The main function of the module is to process user requests, including communication of upper layer computing jobs or business application programs (representing possible application programs calling the interface). First, the user access request is processed, then the task is issued to the data interaction module, and finally the processing result is returned to the upper layer user.

[0044] The data interaction module obtains the file path name, data storage location and creation time through the operation request statement.

[0045] The underlying storage system comprises a file system and an object storage system.

[0046] The data interaction module is initialized based on a configuration file, accesses the file system or the storage system, and registers a unique identifier for each underlying storage system according to the data storage location.

[0047] The creation time is reserved information of the file, and the file creation time is queried.

[0048] The data interaction module accesses the metadata management module through the file path name to obtain the underlying storage system corresponding to the file path.

[0049] ​​The data interaction module converts the operation request statement into an operation statement of the underlying storage system corresponding to the file path through the file system adapter, and returns the operation result to the client.

[0050] Specifically, the data interaction module is configured to perform data operations on the underlying storage system thereof and respond to requests from the client.

[0051] Based on the configuration file initialization, one or more file systems or object storage systems are accessed, and a unique identifier is registered for each storage system. The configuration file is configured by the user during installation of the system, and the configuration file mainly includes the link of the storage system to be accessed.

[0052] After receiving the request from the client, the request information is analyzed, and the metadata management module is interacted to obtain the metadata information of the requested data, including the file path name (the file path name is used to access the corresponding file in the actual data storage system), the data storage location (the unique identifier of the storage system), the creation time (the creation time is the file retention information, which is used to query the file creation time), and the like.

[0053] The requested statement is adapted according to the type of the storage system through the file system adapter, and the underlying storage system is requested to perform the corresponding data operation, and finally the execution result is returned to the client. The request is the operation request statement parsed from the user request information by the client module.

[0054] The client returns the operation result to the user.

[0055] The metadata management module manages the storage system metadata and the data file metadata. The data file metadata includes structured data and unstructured data. The storage system metadata is related data of all underlying storage systems accessed by the system, and subsequent operation requests are accessed to the underlying storage system for operation through the related data.

[0056] Specifically, the metadata management module is configured to manage global metadata, including storage system metadata, structured data, and unstructured data metadata. The structured data and the unstructured data are data file metadata. The storage system metadata is related data of all underlying storage systems accessed by the system, and subsequent operation requests are accessed to the specific storage system for operation through the information. The data stored is kept globally unique in path to ensure the correctness and uniformity of data access.

[0057] The file system adapter performs a unified program interface for structured data and unstructured data.

[0058] The file system adapter performs free conversion between file systems and object storage systems:

[0059] The object storage system performs a uniform program interface operation of the file system: limiting the namespace operation under the same bucket space, taking the bucket as the root directory of the file system, and the subsequent file or directory creation behavior is to create a key under the directory of the file system; taking the absolute path of the file or directory as the key, and the value as the written file content, to perform file system mode writing and reading; after conversion, the file system lists all files under the directory, and performs prefix query on the entire key table.

[0060] The file tree system performs a key-value storage design mode: corresponding to the object storage system performing a uniform program interface operation of the file system, the key in the object storage system is taken as the absolute path of the file; creating a key-value pair is to store the value of the file content under the key path in the file system.

[0061] Specifically, the file system adapter implements the uniform API (uniform program interface) of structured data and unstructured data. API is the full name of Application Program Interface in English, and the Chinese translation is application program interface.

[0062] As shown in Figure 2 and Figure 3 , it is a function diagram of the file system adapter module, which shows how to correspond the operation instructions of the file system and the operation instructions of the object storage to each other and the conversion between the namespaces of the two.

[0063] To realize the free conversion between the traditional file system and the object storage system structure, there are mainly two points:

[0064] The first point: the object storage system implements the API operation of the traditional file system: limiting the namespace operation under the same bucket space, that is, taking the bucket as the root directory of the file system. The subsequent file or directory creation behavior is to create a key under the bucket directory (this part is the namespace converter). Taking the absolute path of the file or directory as the key, and the value as the written file content (the value is empty in the case of the directory as the key), that is, the file system mode writing and reading are realized. After conversion by this mode, the file system also supports listing all files under the directory, that is, prefix query is realized on the entire key table (this part is the operation instruction converter).

[0065] The second point: the file tree system implements the key-value storage design mode: corresponding to the above conversion mode (the object storage system implements the API operation of the traditional file system described in the first point), the key in the object storage system is taken as the absolute path of the file (this part is the namespace converter). Creating the Key-Value (key-value pair) is to store the file content of the Value under the Key path in the file system (this part is the operation instruction converter).

[0066] The embodiment of the present application also discloses a hybrid storage method and a hybrid storage system, comprising the following steps.

[0067] The client step is to access a user request in real time, parse the user request, and extract a user operation request statement.

[0068] The processing step is to convert the operation request statement into an operation statement of a corresponding underlying storage system to perform operation and obtain an operation result.

[0069] The returning step is to return the operation result to the user.

[0070] Specifically, the following steps are included.

[0071] Step 1: The client module accesses a user request in real time, parses the user request, and extracts a user operation request statement.

[0072] Step 2: The data interaction module obtains a file path name in the operation request statement.

[0073] Step 3: The data interaction module accesses the metadata management module through the file path name to obtain an underlying storage system corresponding to the file path.

[0074] Step 4: The data interaction module converts the operation request statement into an operation statement of the underlying storage system through the file system adapter to perform operation and returns the operation result to the client module.

[0075] Step 5: The client module returns the operation result to the user.

[0076] The embodiment of the present application also discloses a hybrid storage medium comprising the hybrid storage system. The medium is a carrier of data storage.

[0077] Based on the above method, the free conversion of the file system tree structure to the key-value mode of the object storage system is realized. That is, the unstructured data such as audio and video under the object storage system and the structured data under the file system can be directly accessed without the need of business code adaptation. By adopting the mapping of the tree structure of the file system to the key-value structure, the unified access demand of the upper-layer business such as machine learning and data mining for the structured data and the unstructured data is solved, and the business implementation process is simplified.

[0078] The free conversion of the multi-level tree structure of the traditional file system to the key-value structure of the object storage system provides a compatible data access interface for the upper-layer computing engine and machine learning, shrinks the underlying storage system access interface, shields the data structure difference, effectively avoids the data island, and simplifies the business end-to-end process.

[0079] Those skilled in the art know that, in addition to implementing the system provided by the present application and each device, module and unit thereof in the form of pure computer readable program code, the system provided by the present application and each device, module and unit thereof can also be implemented in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers and embedded microcontrollers, etc. by logically programming the method steps to achieve the same functions. Therefore, the system provided by the present application and each device, module and unit thereof can be considered as a hardware component, and the devices, modules and units included therein for achieving various functions can also be considered as structures within the hardware component; the devices, modules and units for achieving various functions can also be considered as both software modules implementing methods and structures within hardware components.

[0080] The specific embodiments of the present application are described above. It needs to be understood that the present application is not limited to the specific embodiments described above, and various changes or modifications can be made by those skilled in the art within the scope of the claims, which does not affect the essential content of the present application. The embodiments of the present application and the features in the embodiments can be combined with each other in any manner without conflict.

Claims

1. A hybrid storage system, characterized by, The system comprises a client, a data interaction module, a file system adapter and an underlying storage system. The client accesses user requests in real time, parses the user requests and extracts user operation request statements. The data interaction module converts the operation request statements into operation statements of the corresponding underlying storage system through the file system adapter, performs operations and returns operation results to the client. The client returns the operation results to the user. The system further comprises a metadata management module. The data interaction module obtains a file path name through the operation request statement. The data interaction module accesses the metadata management module through the file path name to obtain the underlying storage system corresponding to the file path. The data interaction module converts the operation request statement into an operation statement of the underlying storage system corresponding to the file path through the file system adapter, performs operations and returns operation results to the client. The file system adapter freely converts the structures of the file system and the object storage system: The object storage system performs unified program interface operations of the file system: limits namespace operations under the same bucket space, takes the bucket as the root directory of the file system, and creates keys under the directory of the file system in subsequent file or directory creation behaviors; takes the absolute path of the file or directory as the key, and takes the written file content as the value to perform file system mode writing and reading; after conversion, the file system lists all files under the directory, and performs prefix query on the entire key table. The file tree system performs key-value storage design mode: corresponding to the unified program interface operation of the file system of the object storage system, the key in the object storage system is taken as the absolute path of the file; creating a key-value pair stores the value of the file content under the key path in the file system.

2. The hybrid storage system of claim 1, wherein, The client provides a unified data access service interface for the user and processes user requests.

3. The hybrid storage system of claim 1, wherein, The data interaction module obtains a data storage location through the operation request statement. The underlying storage system comprises a file system and an object storage system. The data interaction module is initialized based on a configuration file, accesses the file system or the storage system, and registers a unique identifier for each underlying storage system according to the data storage location.

4. The hybrid storage system of claim 1, wherein, The data interaction module obtains a creation time through the operation request statement, retains information for the file, and queries the file creation time.

5. The hybrid storage system of claim 1, wherein, The metadata management module manages storage system metadata. The storage system metadata is related data of all accessed underlying storage systems, and subsequent operation requests are accessed to the underlying storage system for operation through the related data.

6. The hybrid storage system of claim 1, wherein, The metadata management module manages data file metadata. The data file metadata comprises structured data and unstructured data. The file system adapter performs unified program interfaces for structured data and unstructured data.

7. A method of hybrid storage, characterized by, The system of any one of claims 1-6 comprises the following steps: The client step: accessing user requests in real time, parsing the user requests and extracting user operation request statements. The processing step: converting the operation request statements into operation statements of the corresponding underlying storage system to perform operations and obtain operation results. The return step: returning the operation results to the user.

8. A hybrid storage medium characterized by A hybrid storage system comprising any of claims 1-6.

Images