A method and device for constructing a hash index of a distributed storage system

Abstract

The present disclosure provides a distributed storage system hash index construction method and device, relates to the field of artificial intelligence, in particular to the technical field of cloud computing, cloud storage and cloud database, and can be applied to an intelligent cloud scene. The specific implementation scheme is as follows: a file used for storing memory data is created, and a corresponding hash index of the file is constructed; the information type of the file is obtained; and the index value of the hash index is constructed according to the information type. Through the present disclosure, the performance of the storage system can be improved, and the competitiveness of cloud services can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, particularly to the fields of cloud computing, cloud storage, and cloud databases, and can be applied in intelligent cloud scenarios. Specifically, it relates to a method, apparatus, electronic device, storage medium, and product for constructing a hash index for a distributed storage system. Background Technology

[0002] Distributed file systems can provide users with secure and scalable file storage services. Through standard file access protocols, they provide unlimited scalability, high reliability, and high performance file storage capabilities for computing resources such as virtual machines and containers in the cloud.

[0003] As the size of file systems increases, the performance requirements of distributed file storage systems also become increasingly demanding. Summary of the Invention

[0004] This disclosure provides a method, apparatus, electronic device, storage medium, and product for constructing a hash index in a distributed storage system.

[0005] According to a first aspect of this disclosure, a method for constructing a hash index in a distributed storage system is provided, comprising:

[0006] Create a file to store the data in memory, and build a corresponding hash index for the file; obtain the information type of the file; and build the index value of the hash index based on the information type.

[0007] According to a second aspect of this disclosure, an apparatus for constructing a hash index in a distributed storage system is provided, the apparatus comprising:

[0008] A creation module is used to create a file for storing memory data and to build a corresponding hash index for the file; an acquisition module is used to acquire the information type of the file; and a construction module is used to construct the index value of the hash index based on the information type.

[0009] According to a third aspect of this disclosure, an electronic device is provided, comprising:

[0010] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of the first aspect or the second aspect.

[0011] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are configured to cause the computer to perform the method according to the first or second aspect.

[0012] According to a fifth aspect of this disclosure, a computer product is provided, including a computer program that, when executed by a processor, implements the method according to the first or second aspect.

[0013] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0014] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0015] Figure 1 A flowchart illustrating a method for constructing a hash index in a distributed storage system according to an embodiment of this disclosure is shown.

[0016] Figure 2 A flowchart illustrating a method for obtaining files corresponding to memory data according to an embodiment of this disclosure is shown.

[0017] Figure 3 A flowchart illustrating a method for creating a file for storing memory data, according to an embodiment of this disclosure, is shown.

[0018] Figure 4 This diagram illustrates the structure of a device for constructing a hash index in a distributed storage system according to an embodiment of the present disclosure.

[0019] Figure 5 A schematic block diagram of an example electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0020] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0021] Distributed file systems provide users with secure and scalable file storage services. Through standard file access protocols, they offer virtually unlimited scalability, high reliability, and high performance file storage capabilities for computing resources such as virtual machines and containers in the cloud. As the scale of file systems increases, the performance requirements of distributed file storage systems also become more demanding, especially during data retrieval.

[0022] In related technologies, indexes in distributed storage systems are mainly used to store the mapping from user logical data to physical data. The smallest unit of a typical index is 4k, and a common index structure is a tree index, such as a Log-Structured-Merge-Tree (LSM tree). An LSM tree is a multi-level storage method where data access may occur in a memtable or on an N-level sorted string table (SST) on disk.

[0023] However, the related technologies suffer from read amplification and write amplification during data file compression, resulting in low disk utilization.

[0024] To address the aforementioned technical issues, this disclosure provides a method and apparatus for constructing a hash index for a distributed storage system. By constructing corresponding hash indexes for files on the disk of a distributed storage system, the performance of the storage system can be improved, thereby enhancing the competitiveness of cloud services, while ensuring high reliability and high availability.

[0025] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0026] Figure 1 This illustration shows a flowchart of a method for constructing a hash index in a distributed storage system according to an embodiment of the present disclosure. Figure 1 As shown, the method may include:

[0027] In step S110, a file is created to store the memory data, and a corresponding hash index is constructed for the file.

[0028] In this embodiment of the disclosure, the distributed storage system can be used to store unstructured data objects such as images and videos, including Blob objects, fixed-length blocks, and large files.

[0029] The embodiments provided in this disclosure are applicable to single-machine storage and / or data retrieval, and can of course be used in other scenarios as well.

[0030] In this embodiment, the operating system can map and write memory data to files based on the data mapping relationship. Furthermore, a corresponding hash index can be built for each file.

[0031] In step S120, the information type of the file is obtained.

[0032] In this embodiment of the disclosure, mapping memory data to a corresponding file allows determination of the information type of that file. The information type can be attribute information, data information, or similar types.

[0033] In step S130, the index value of the hash index is constructed according to the information type.

[0034] In this embodiment of the disclosure, an index value corresponding to the hash index is constructed for the file based on the information type of the file.

[0035] The method for constructing hash indexes in a distributed storage system provided in this disclosure creates files for in-memory data and maps the in-memory data to the corresponding files, reducing the overhead of data serialization. Constructing corresponding hash indexes for files on the disk of the distributed storage system improves the performance of the storage system and enhances the competitiveness of cloud services while ensuring high reliability and availability. Furthermore, data access via hash indexes is fast and does not involve additional read / write amplification, significantly improving the performance of the storage system.

[0036] In this disclosure, a corresponding hash index can be constructed for each replica data file in a distributed storage system. The same data can be distributed and stored on different storage devices.

[0037] Figure 2 This illustration shows a flowchart of a method for obtaining files corresponding to memory data according to an embodiment of this disclosure. Figure 2 As shown, the method may include:

[0038] In step S210, at least one copy of the memory data in the distributed storage system is obtained.

[0039] In step S220, the file containing the slice data corresponding to each copy of the data is obtained.

[0040] In this embodiment of the disclosure, the same memory data can be stored in different storage devices in the distributed storage system, and the memory data stored in each storage device is a copy of the data.

[0041] In this disclosure, the replica data may also be referred to as a replication group. Therefore, the backend of a distributed storage system consists of multiple replication groups.

[0042] In this embodiment, multiple data slices corresponding to each copy of data can also be obtained, thereby obtaining the file corresponding to each data slice on the disk, and constructing the index value of the corresponding hash index for each file.

[0043] In this embodiment of the disclosure, a hash index can be built for each file.<key,value> ), and based on the file's information type<key,value> Assign a value.

[0044] In one embodiment of this disclosure, in response to the information type being attribute information, the file's inode is used as the key of the hash index, and the file's attributes are used as the index value of the hash index. The attribute information may include file size, storage time in memory, and other attribute information.

[0045] Furthermore, if the information type is attribute information, the key can be the index node of a hash index. <inode>The value is a pointer to disk data. Inodes store basic information about files and directories, including time, filename, user, and group.

[0046] In another embodiment of this disclosure, in response to the information type being data information, the inode and its data offset of the file are used as the key of the hash index, and the storage name of the file is used as the index value of the hash index. Here, the data information can be the specific content of memory data; for example, if the data content is ABCD, then the data information can be ABCD.

[0047] Furthermore, if the information type is data information, the key can be a hash.<inode+extent_index> The inode is the file number, the extent_index is the data offset, and the value stores the block name (blockid) of this part of the data. This blockid can represent the offset and length of the local disk, or it can be a storage ID of another system.

[0048] Figure 3 This illustration shows a flowchart of a method for creating a file for storing memory data, as provided in an embodiment of this disclosure. Figure 3 As shown, the method may include:

[0049] In step S310, storage space is requested based on the memory data.

[0050] In step S320, a file is created, data in the storage space is mapped to the file, and the relevant metadata of the memory data is written into the file header information.

[0051] In this embodiment, the operating system can request storage space to control the disk to open corresponding files. Then, an initialization function (e.g., the memset function) is used to set all the contents of a certain memory block to specified values, mapping the memory data to the file. The relevant metadata of the memory data is then written to the file header.

[0052] In this embodiment of the disclosure, after constructing a hash index for the file corresponding to the memory data, the required memory data can be quickly found based on the hash index.

[0053] In this embodiment, a request can be initiated to the backend of the distributed storage system based on the hash index value, thereby accessing the hash index table and searching for the corresponding memory data on the disk. If the requested index value exists in the index table, the data on the disk is accessed; otherwise, the data is empty.

[0054] By constructing a hash index for the disk file corresponding to the memory data in this embodiment, better read performance is achieved, no additional write amplification is required, and the overhead of data serialization is eliminated.

[0055] Based on and Figure 1 The method shown follows the same principle. Figure 4 This illustration shows a schematic diagram of the structure of a hash index construction apparatus for a distributed storage system provided in an embodiment of the present disclosure, as shown below. Figure 4 As shown, the hash index construction apparatus 400 of the distributed storage system may include:

[0056] The creation module 401 is used to create a file for storing memory data and to build a corresponding hash index for the file; the acquisition module 402 is used to acquire the information type of the file; and the construction module 403 is used to construct the index value of the hash index according to the information type.

[0057] In this embodiment of the disclosure, the acquisition module 402 is used to acquire at least one copy of the memory data in the distributed storage system; and to acquire the file where the slice data corresponding to each copy of the data is located.

[0058] In this embodiment of the disclosure, the construction module 403 is configured to, in response to the information type being attribute information, use the file's index node as the key of the hash index and the file's attribute as the index value of the hash index.

[0059] In this embodiment of the disclosure, the construction module 403 is configured to, in response to the information type being data information, use the file's index node and its data offset as the key of the hash index, and use the file's storage name as the index value of the hash index.

[0060] In this embodiment of the disclosure, the creation module 401 is used to apply for storage space based on the memory data; create a file, map the data in the storage space to the file, and write the relevant metadata of the memory data into the file header information.

[0061] In this embodiment of the disclosure, the device further includes a lookup module 404;

[0062] The search module 404 is used to search for memory data in the distributed storage system based on the index value.

[0063] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0064] In an exemplary embodiment, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described in the above embodiments. The electronic device may be the computer or server described above.

[0065] In an exemplary embodiment, the readable storage medium may be a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method described in the above embodiments.

[0066] In an exemplary embodiment, the computer program product includes a computer program that, when executed by a processor, implements the method described in the above embodiments.

[0067] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0068] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0069] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0070] like Figure 5 As shown, device 500 includes a computing unit 501, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 502 or a computer program loaded from storage unit 508 into random access memory (RAM) 503. RAM 503 may also store various programs and data required for the operation of device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. Input / output (I / O) interface 505 is also connected to bus 504.

[0071] Multiple components in device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0072] The computing unit 501 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as the method for constructing a hash index for a distributed storage system. For example, in some embodiments, the method for constructing a hash index for a distributed storage system can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the method for constructing a hash index for a distributed storage system described above can be performed. Alternatively, in other embodiments, computing unit 501 may be configured by any other suitable means (e.g., by means of firmware) to perform a method for constructing a hash index for a distributed storage system.

[0073] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0074] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0075] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0076] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0077] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0078] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0079] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0080] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.< / inode>

Claims

1. A method for constructing a hash index in a distributed storage system, comprising: Create a file to store the memory data, and build a corresponding hash index for the file; Obtain the information type of the file, wherein the information type is attribute information or data information; Construct the index value of the hash index based on the information type; Wherein, constructing the index value of the hash index according to the information type includes: In response to the information type being data information, the file's inode and its data offset are used as the key of the hash index, and the file's storage name is used as the index value of the hash index; In response to the information type being attribute information, the file's inode is used as the key of the hash index, and the file's attribute is used as the index value of the hash index.

2. The method according to claim 1, wherein, The file corresponding to the memory data to be acquired includes: Obtain at least one copy of the in-memory data in the distributed storage system; Obtain the file containing the slice data corresponding to each of the aforementioned copy data.

3. The method according to claim 1, wherein, Creating the file for storing the memory data includes: Request storage space based on the memory data; Create a file, map the data in the storage space to the file, and write the relevant metadata of the memory data into the file header information.

4. The method according to claim 1, wherein, After constructing the index value of the hash index based on the information type, the method further includes: Based on the index value, locate the memory data in the distributed storage system.

5. An apparatus for constructing a hash index in a distributed storage system, the apparatus comprising: A module is created to create files for storing in-memory data and to build a corresponding hash index for those files. The acquisition module is used to acquire the information type of the file, wherein the information type is attribute information or data information; A construction module is used to construct the index value of the hash index based on the information type; The building module is used for: In response to the information type being data information, the file's inode and its data offset are used as the key of the hash index, and the file's storage name is used as the index value of the hash index; In response to the information type being attribute information, the file's inode is used as the key of the hash index, and the file's attribute is used as the index value of the hash index.

6. The apparatus according to claim 5, wherein, The acquisition module is used for: Obtain at least one copy of the in-memory data in the distributed storage system; Obtain the file containing the slice data corresponding to each of the aforementioned copy data.

7. The apparatus according to claim 5, wherein, The creation module is used for: Request storage space based on the memory data; Create a file, map the data in the storage space to the file, and write the relevant metadata of the memory data into the file header information.

8. The apparatus according to claim 5, wherein, The device also includes a search module; The search module is used to search for memory data in the distributed storage system based on the index value.

9. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-4.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-4.

11. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-4.

Images