A mass object fast storage system based on NAS file system

Abstract

The application discloses a kind of mass object fast storage systems based on NAS file system, it is related to big data service and so on field.The mass object fast storage system of the application includes server module, client module and NAS file system;After client module receives the data that user writes, object data is assembled into byte array and is sent to server module;Server module calculates the file name of byte array writing NAS system, file name is universal unique identification code uuid, array is stored in the file of corresponding NAS system, and file name and the index position of array in file are returned to client;Client calculates the final unique identification of each object, and user reads object file from NAS system using final identification.The application can store a large number of object data based on NAS fast, avoid metadata problem, and can provide super-high object writing speed close to network card limit.

Description

Technical Field

[0001] This invention belongs to the field of Internet and cloud computing, big data services, specifically referring to a high-speed storage system for massive objects based on NAS (Network Attached Storage) file system. Background Technology

[0002] With the development of the internet, data is growing explosively, leading to a demand for massive storage systems. NAS (Network Attached Storage) is a device connected to a network that provides data storage capabilities; it is also known as a "network storage device." NAS is a dedicated data storage server that centers on data, completely separating storage devices from servers and centrally managing data. This frees up bandwidth, improves performance, reduces total cost of ownership, and protects investment. The cost of NAS is far lower than using server storage, while its efficiency is far higher, and it can be used as a file system.

[0003] Massive data storage involves object data, a concept provided by Amazon Simple Storage Service (S3) object storage service. Object data refers to a single piece of data that is stored, consisting of objects and metadata. An object can be any sequence of bytes, such as "This is a test file," while metadata is a set of key-value pairs describing the object, such as "Creation time = 2022-01-02 12:00:00, Creator = Zhang San, Location = Beijing."

[0004] Conventional storage systems, including NAS, require accessing the metadata of actual data before they can access the data itself. This metadata is recorded in inodes (index nodes). When storing files, in addition to file size being limited by disk space, the number of files is also constrained by inodes. With a massive number of small files, a significantly larger amount of metadata will inevitably be generated. If the files are too small, but the number of files reaches the inode limit, a situation will arise where the disk is not full, but the inodes are exhausted. In this case, no new files can be written, resulting in wasted storage space. This further amplifies the problems of poor file system scalability and low retrieval efficiency. For example, traditional NAS file systems experience performance bottlenecks when the number of files reaches tens of millions, severely impacting business efficiency. Due to the significant challenges in metadata management, storage performance, and access efficiency, storing massive amounts of small files has become a recognized industry challenge.

[0005] In addition, in a conventional file system, a file can only store a few fixed metadata, such as creation time and creator, and cannot store additional metadata, such as data source, location, organization, category, etc. Summary of the Invention

[0006] This invention addresses the issues of poor performance, limited file storage capacity, and inability to store additional metadata in scenarios involving massive amounts of small files using NAS. It provides a high-speed, massive object storage system based on a NAS file system, capable of rapidly storing large amounts of object data on NAS, thus meeting the needs of enterprises in such scenarios.

[0007] This invention discloses a high-speed, massive object storage system based on a NAS file system, comprising: a server module, a client module, and a NAS file system; wherein:

[0008] After receiving the data written by the user, the client module assembles the data and sends it to the server module. The data is assembled as follows: the object data written by the client is assembled into a byte array B. Each object data consists of four parts: key-value pair data length, key-value pair data, object data length, and object data. The key-value pair data records the metadata in the object. Each metadata is stored by a key-value pair. Each key-value pair consists of key length, key, value length, and value.

[0009] After receiving the byte array B, the server module calculates the filename to be written to the NAS file system. The filename is named using a universally unique identifier (uuid), which contains timestamp information. The server module directly writes the byte array B to the NAS file system. The file path is: / nas_path / year / month / day / hour / uuid, where " / year / month / day / hour" in the file path is calculated from the timestamp in the uuid. At the same time, the server module returns the filename and the index position of the byte array B in the NAS file to the client module.

[0010] After receiving the filename (uuid) and index position in the NAS file from the byte array B, the client module calculates the index position (index2) of each object in the NAS file by combining the index position of each object in the byte array B, obtains the final unique identifier (uuid_index2) of the object, and outputs it to the user for storage.

[0011] The file name UUID generated by the server module contains the current time information.

[0012] The server module receives the object identifier uuid_index2 sent by the user, parses the time information from the uuid, obtains the path of the object in the NAS file system, and then reads the object data from the file at the corresponding path according to index2, and returns it to the client module in the form of a byte array. The client module parses the object information and descriptive metadata from the byte array.

[0013] Compared with the prior art, the advantages and positive effects of the present invention are as follows:

[0014] Traditional object storage systems require a metadata system to record the location of objects written by clients on the hard drive / NAS. However, since most object data is relatively small, the amount of metadata can be enormous, resulting in significant storage space consumption. Furthermore, queries also require reading metadata, necessitating frequent sorting and organization of the metadata during object data writing. Maintaining a large metadata system consumes substantial system resources, especially in distributed environments. Maintaining data consistency requires locking at different levels, leading to slow data writing. This invention's massive object fast storage system avoids metadata issues by introducing an automatically generated unique object identifier (uuid_index2), thus providing ultra-high object write speeds approaching network card limits, far exceeding any other object storage system on the market. In this invention's massive object fast storage system, the object's unique identifier (uuid_index2) is generated programmatically, avoiding both the storage of its own metadata and the need for users to generate object identifiers themselves. This prevents data overwriting and loss due to unique identifier conflicts caused by improper use. Furthermore, this system can also reduce the overhead of opening and destroying network connections by caching multiple network requests into a single request through client-side caching, thereby improving write speed. Attached Figure Description

[0015] Figure 1 This is a schematic diagram illustrating the use of the snowflake algorithm to generate file name UUIDs in an embodiment of the present invention;

[0016] Figure 2 This is an implementation diagram of the high-speed storage system for massive objects based on the NAS file system of the present invention;

[0017] Figure 3 This is an example diagram illustrating the storage of write object data using the system of this invention. Detailed Implementation

[0018] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0019] This invention provides a high-speed, massive object storage system based on a NAS file system, comprising a client module, a server module, and a NAS file system. The NAS file system is used to store the final data.

[0020] The massive object fast storage system of this invention avoids metadata problems by introducing a method to automatically generate unique identifiers for objects. For example... Figure 1As shown, this embodiment of the invention uses the snowflake algorithm to generate the file's UUID (Universally Unique Identifier). Then, combined with the object's index position (index2) in the NAS file, the object's final unique identifier, uuid_index2, is generated.

[0021] like Figure 2 As shown, after receiving data written by the user, the client module assembles the data and sends it to the server module. For example, it assembles a set of data written by the user into a byte array B1, where array B1 contains one or more object data, labeled as b1, b2, etc. The server module directly writes the assembled data to the NAS file system and returns the file name and the offset of the data in the NAS file system to the client. The client module calculates the final unique identifier of each object based on the file name and the offset of the file in the NAS system returned by the server module and returns it to the user for storage.

[0022] The client module assembles data in the following way: (e.g.) Figure 2 As shown in (b), the client module receives a set of data written by the user, assembles the data into a byte array B, which consists of multiple object data b; each object data b consists of four parts, recording: the length of the key-value pair data, the key-value pair data, the length of the object data, and the object data; as shown in (b). Figure 2 As shown in (c), the key-value pair data c records the metadata of the object. There are N metadata items, where N is a natural number. Each metadata item is stored as a key-value pair. A single key-value pair consists of four parts: key length, key, value length, and value. Each object data can be compared to a small file, and a group of data consisting of multiple objects is a large file. For example, a byte array B is likened to a large file, and an object data b is likened to a small file.

[0023] like Figure 3 As shown, for example, an object data b1 is written, the content of which is a byte sequence: This is a testfile; the metadata describing the object includes two key-value pairs: name = zhangsan; age = 19. Object data b1 contains: key-value pair data length = 25, key-value pair data contains two key-value pairs, object data length = 19, object data = This is a test file. The two key-value pairs are: key length = 4, key = name, value length = 8, value = zhangsan; key length = 3, key = age, value length = 2, value = 2.

[0024] The client module assembles the written data into a byte array B, and records the index position bIndexN of each object in byte array B, for example, the index position bIndex1 of object b1 and the index position bIndex2 of object b2. Then, it sends the byte array B to the server. bIndexN refers to the offset of the first byte in byte array B when the object data is assembled.

[0025] After receiving the byte array B, the server module directly writes it to the NAS file system and returns the filename and index position (aIndex) of the byte array B within the NAS file to the client. The filename for the byte array B written to the NAS file system is called a UUID, which is a unique identifier generated for the file; the index position (aIndexN) refers to the offset of the first byte of array B in the file when it is written to the NAS file. The file path where the server module writes the byte array B to the NAS system is: / nas_path / year / month / day / hour / uuid.

[0026] like Figure 1 As shown, the file name UUID written to the NAS in this embodiment of the invention is generated using the Snowflake algorithm. The UUID is a long type, consisting of 64 bits. The first bit indicates the sign of the UUID, and in this embodiment, the first bit is set to 0; the second to fourth bits represent the timestamp information; the fourth to fifth bits represent the machine ID; the fourth to fifth bits represent the service ID, and in this embodiment, the fourth bit is set to 00000; and the fifth to sixth bits represent the sequence number.

[0027] The machine ID is fixed during cluster initialization and will not change. Each NAS node has a different machine ID, such as the machine ID of the second node being 00001.

[0028] A timestamp is the number of milliseconds between the current time and the system initialization time. For example, if the system was initialized and deployed on 2023-03-03 12:00:00000, and the current time is 2023-03-04 12:00:00 000, which is a full day apart from the system initialization time, the corresponding timestamp value is 24*60*60*1000=86400000, which is 101001001100101110000000000.

[0029] The 53rd to 64th bits are used to distinguish different file names under the same millisecond timestamp on the same machine. A unique UUID can be generated by incrementing the sequence number.

[0030] like Figure 1As shown in the figure, a 64-bit UUID generated by the snowflake algorithm for the byte array B in this embodiment of the invention is as follows: 0 0000000000000010100100110010111000000000 00001 00000 0000000000000

[0032] The corresponding decimal conversion is: 362387865600000.

[0033] The file path where byte array B is written to the NAS system is: naspath / 2023 / 03 / 04 / 12 / 362387865600000, which is the storage path of byte array B on the NAS, calculated and uniquely located using its UUID. The index position of byte array B in the NAS file system is its offset aIndexN within the file naspath / 2023 / 03 / 04 / 12 / 362387865600000, for example... Figure 2 In the array, the index position of byte array B1 is aIndex1, and the index position of array B2 is aIndex2.

[0034] The client module stores the UUID of the byte array B returned by the storage client module and its index position aIndexN on the NAS. Combined with the index positions bIndexN of each object in the byte array B calculated by the client module, the index position index2 of each object in the NAS file is calculated and returned to the user as the unique identifier of the small file, which is then saved by the user.

[0035] For example, a user writes two objects to the client, which assembles the data into array B. The first object has an offset of 0 at bIndex1 in array B, and the second object has an offset of 420 at bIndex2. The client sends array B to the server, which writes the byte array B to a file in the NAS file system at the path ' / naspath / 2023 / 03 / 04 / 12 / 362387865600000'. The first byte of the byte array B has an index position of 108341200 at aIndex1 in the file. The server then returns the UUID: 362387865600000 and aIndex1: 108341200 to the client. The client can calculate the absolute offset of the first object in the file ' / nas path / 2023 / 03 / 04 / 12 / 362387865600000' as "absolute offset of byte array B in the file + relative offset of object 1 in byte array B" = 108341200 + 0 = 108341200; the absolute offset of the second object in the file ' / nas path / 2023 / 03 / 04 / 12 / 362387865600000' as "absolute offset of byte array B in the file + relative offset of object 2 in byte array B" = 108341200 + 420 = 108341620. The client then obtains the object's final unique identifier as "uuid_index2", where index2 is the object's index position in the NAS file. For example, the final unique identifier of the first object is "362387865600000_108341200", and the final unique identifier of the second object is "362387865600000_108341620".

[0036] When a user requests object data, they provide `uuid_index2` to the server module for querying. The server module parses the `uuid`, extracts the time information, and calculates the storage path of the large file on the NAS: ` / nas / year / month / day / hour / uuid`. Based on the `index2` information, it reads the object-related information from that NAS file and returns it to the client module as a byte array. The client module then parses the byte array to extract the object's information (`data`) and descriptive information (`para`).

[0037] The method of this invention sets object data and key-value pairs using a length-value approach. When locating an index position, the length is read first, and then the corresponding bytes (key-value pairs) are read based on that length. For example, when reading byte 108341200 of the file / naspath / 2023 / 03 / 04 / 12 / 362387865600000, the number 25 is read. Then, the next 25 bytes are read. The first byte read is 4 bytes long, so 4 more bytes are read to get the name. This process continues until the key-value pair 'zhangsan' is obtained, along with the age and 19, and finally the object text 'This is a test file'.

[0038] The server module also sets a maximum threshold for the size of large files stored in the NAS file system, such as 1GB. When the server module detects that the file being written to the NAS file system is larger than this threshold, it will regenerate a file name UUID according to the snowflake algorithm, close the previous large file, and start writing data to the new UUID file.

[0039] Except for the technical features described in the specification, all other technologies are known to those skilled in the art. Descriptions of well-known components and technologies are omitted in this invention to avoid redundancy and unnecessary limitation. The embodiments described above do not represent all embodiments consistent with this application. Various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of this invention are still within the protection scope of this invention.

Claims

1. A high-speed, massive object storage system based on a NAS file system, comprising a server module, a client module, and a NAS file system; wherein: After receiving data written by the user, the client module assembles the data and sends it to the server module. The data assembly process involves assembling the object data written by the client into a byte array B. Each object data item consists of four parts: key-value pair data length, key-value pair data, object information length, and object information. The key-value pair data records descriptive metadata of the object, with each metadata item stored as a key-value pair. Each key-value pair consists of a key length, a key, a value length, and a value. The client module also records the index position of each object in the byte array B. After receiving the byte array B, the server module calculates the filename to be written to the NAS file system. The filename is named using a universally unique identifier (uuid), which contains timestamp information. The server module directly writes the byte array B to the NAS file system. The file path is: / nas_path / year / month / day / hour / uuid, where " / year / month / day / hour" in the file path is calculated from the timestamp in the uuid. At the same time, the server module returns the filename and the index position of the byte array B in the NAS file system to the client module. After receiving the file name (uuid) and its index position in the NAS file from the byte array B, the client module combines the index position of each object in the byte array B to calculate the index position (index2) of each object in the NAS file, obtains the final unique identifier (uuid_index2) of the object, and outputs it to the user for storage. The server module receives the object identifier uuid_index2 sent by the user, parses the time information from the uuid, obtains the path of the object in the NAS file system, and then reads the object data from the file at the corresponding path according to index2, and returns it to the client module in the form of a byte array. The client module parses the object information and descriptive metadata from the byte array.

2. The high-speed storage system for massive objects based on a NAS file system according to claim 1, characterized in that, The server-side module uses the snowflake algorithm to generate filename UUIDs, including: First, generate a 64-bit UUID: the first bit is set to 0; bits 2 to 42 record the timestamp, which is the current time in binary relative to the system initialization time in milliseconds; bits 43 to 47 represent the machine ID, which is different for different NAS nodes; bits 48 to 52 are set to 00000; bits 53 to 64 represent the sequence number, used to distinguish different file names under the same millisecond timestamp on the same machine. Next, the generated 64-bit UUID is converted to decimal, and the resulting decimal UUID is used as the file name.

3. The high-speed storage system for massive objects based on a NAS file system according to claim 1, characterized in that, The server-side module calculates the index position (index2) of each object in the NAS file, including: Let the index position of byte array B in the NAS file be aIndexN, and let the position of object b in array B be bIndexN. Then the index position of object b in the NAS file is index2 = aIndexN + bIndexN.

Images