Method and apparatus for distributed storage system io real-time aggregation

Abstract

The application provides a method and equipment for IO real-time aggregation of a distributed storage system, comprising: an aggregation object has attributes such as a write name, an allocation size and a reference count, and the final aggregation object file size does not exceed a set size. Before IO writing, the metadata management node is inquired first, for new file writing, a request for generating an aggregation object is sent to an aggregation object generator node to obtain an aggregation object name and a write offset; for the covering IO of the existing file, the write request does not exceed the original size part, and the original aggregation object is written; for the exceeding part, the remaining size request is sent to the aggregation object generator node to obtain a new aggregation object. The IO real-time aggregation method provided by the application can work efficiently in the distributed storage system without using a distributed lock, and greatly reduces the total number of files finally written to the hard disk, and the performance of more than 10 billion file writing is not attenuated in practice.

Description

Method and apparatus for real-time I / O aggregation in distributed storage systems Technical Field

[0001] This invention relates to a method and apparatus for real-time I / O aggregation in a distributed storage system. Background Technology

[0002] In a distributed system environment, data write requests come from different nodes. To achieve real-time IO aggregation, distributed locks are a relatively easy solution to think of. However, this traditional implementation method not only significantly reduces write performance but also increases complexity. Summary of the Invention

[0003] The purpose of this invention is to provide a method and device for real-time I / O aggregation in a distributed storage system.

[0004] To address the above problems, this invention provides a method for real-time I / O aggregation in a distributed storage system, comprising:

[0005] Step S1: Obtain the complete write IO request for writing to a file. The complete write IO request includes: file path, offset, and requested file size, wherein the offset is 0.

[0006] Step S2: Based on the complete write IO request, query the metadata management node to see if the requested write file exists;

[0007] Step S3: If the file does not exist, obtain the write name and write offset of the new aggregate object corresponding to the file to be written.

[0008] Step S4: If the file exists, obtain the write name and write offset of the new aggregate object corresponding to the remaining size of the file to be written, where the remaining size of the file to be written = the size of the file to be written - the size of the original file;

[0009] Step S5: Using the location generator based on the write name of the new aggregate object obtained in step S3, generate the actual write path to the local file system; based on the actual write path to the local file system, the file size, and the write offset of the new aggregate object obtained in step S3, attempt to write the requested file to the hard disk of the corresponding storage node; or, using the location generator based on the write names of the original aggregate object and the new aggregate object obtained in step S4, generate the corresponding actual write paths to the local file system; based on the actual write path to the local file system, the size of the requested file, and the write offset of the new aggregate object obtained in step S4, attempt to write the requested file to the hard disk of the corresponding storage node.

[0010] Step S6: If the file write request is successful, record the file ID, the file size requested, the write name of the new aggregate object, and the write offset of the new aggregate object in the metadata management node.

[0011] Furthermore, in the above method, step S3, if the file does not exist, obtains the write name and write offset of the new aggregate object corresponding to the requested write to the file, including:

[0012] Step S31: If the file does not exist, the aggregation object generator node sends a request to generate a new aggregation object corresponding to the file to be written, and obtains the write name and write offset of the new aggregation object corresponding to the file to be written from the aggregation object generator node based on the request to generate the new aggregation object corresponding to the file to be written. The request to generate the new aggregation object corresponding to the file to be written includes: the file ID to be written and the file size.

[0013] Furthermore, in the above method, step S31, obtaining from the aggregation object generator node the request to generate a new aggregation object corresponding to the application to write the file, and the write name and write offset of the new aggregation object corresponding to the application to write the file, including:

[0014] Step S311: The aggregate object generator node loads the aggregate object collection from the memory file system. Among them, the aggregate object collection The properties of each aggregate object in the document include: write name, allocated size, and reference count;

[0015] Step S312: The aggregation object generator node reads the configuration file information, which includes: the preset upper limit of the aggregation object size, the preset size N of the aggregation object set, the local backend IP address, and the network card MAC address and host name corresponding to the local backend IP address.

[0016] Step S313, if the aggregated object collection If the total number M of aggregated objects in the set is less than the preset size N, then in the set of aggregated objects Continue creating NM aggregate objects, and configure the write name, allocated size and reference count for each newly created aggregate object. The write name of each newly created aggregate object is "HASH(MAC+HostName)" + "128-bit random integer value", the initial value of the allocated size is set to zero, and the initial value of the reference count is set to zero.

[0017] Step S314, from the aggregate object collection Select the aggregate object with the smallest allocated size.

[0018] Step S315: If the allocated size of the selected aggregation object plus the requested write file size is less than or equal to the set aggregation object size value, then the aggregation object is selected;

[0019] Step S316: Update the allocated size and reference count of the selected aggregate object, where the allocated size = the original allocated size of the selected aggregate object + the requested write size, and the reference count = the original reference count of the selected aggregate object + 1.

[0020] Step S317: The original allocated size of the selected aggregate object is used as the write offset of the selected aggregate object. The aggregate object generator node sends a request to generate a new aggregate object corresponding to the file to be written, and feeds back the write offset and write name of the selected aggregate object corresponding to the file to be written.

[0021] Step S318: If the allocated size of the selected aggregate object plus the requested write file size is greater than the set aggregate object size value, then retrieve the aggregate object from the aggregate object collection. Remove the selected aggregate object, and record the name, allocated size and reference count of the removed aggregate object to the metadata management node, then return to step S313 for execution.

[0022] Furthermore, in the above method, step S4, if the file exists, obtains the write name and write offset of the new aggregate object corresponding to the remaining size of the requested file to be written, including:

[0023] Step S41: If the file exists, compare the size of the file to be written with the size of the original file.

[0024] Step S42: If the size of the file to be written is less than or equal to the size of the original file, then the aggregate object of the original file is used;

[0025] Step S43: If the size of the file to be written is greater than the size of the original file, then the portion of the file to be written that is less than or equal to the size of the original file uses the aggregation object of the original file; the remaining size of the file to be written = the size of the file to be written - the size of the original file; by sending a request to the aggregation object generator node to generate a new aggregation object corresponding to the remaining size of the file to be written, wherein the request to generate a new aggregation object corresponding to the remaining size of the file to be written includes: the file ID to be written, and the remaining size of the file to be written; and by obtaining from the aggregation object generator node, based on the request to generate a new aggregation object corresponding to the remaining size of the file to be written, the write name and write offset of the new aggregation object corresponding to the remaining size of the file to be written are fed back.

[0026] Furthermore, in the above method, step S43, obtaining from the aggregate object generator node the write name and write offset of the new aggregate object corresponding to the remaining size of the requested file to be written, based on the request to generate the new aggregate object, includes:

[0027] Step S431: The aggregate object generator node loads the aggregate object collection from the memory file system. Among them, the aggregate object collection The properties of each aggregate object in the document include: write name, allocated size, and reference count;

[0028] Step S432: The aggregation object generator node reads the configuration file information, which includes: the preset upper limit of the aggregation object size, the preset size N of the aggregation object set, the local backend IP address, and the network card MAC address and host name corresponding to the local backend IP address.

[0029] Step S433, if the aggregated object collection If the total number M of aggregated objects in the set is less than the preset size N, then in the set of aggregated objects Continue creating NM aggregate objects, and configure the write name, allocated size and reference count for each newly created aggregate object. The write name of each newly created aggregate object is "HASH(MAC+HostName)" + "128-bit random integer value", the initial value of the allocated size is set to zero, and the initial value of the reference count is set to zero.

[0030] Step S434, from the aggregate object collection Select the aggregate object with the smallest allocated size.

[0031] Step S435: If the allocated size of the selected aggregation object plus the remaining size to be written to the file is less than or equal to the set size value of the aggregation object, then the aggregation object is selected;

[0032] Step S436: Update the allocated size and reference count of the selected aggregate object, where the allocated size = the original allocated size of the selected aggregate object + the remaining size to be written to the file, and the reference count = the original reference count of the selected aggregate object + 1.

[0033] Step S437: The original allocated size of the selected aggregate object is used as the write offset of the selected aggregate object. The aggregate object generator node sends a request to generate a new aggregate object corresponding to the file to be written, and feeds back the write offset and write name of the selected aggregate object corresponding to the file to be written.

[0034] Step S438: If the allocated size of the selected aggregate object plus the remaining size to be written to the file is greater than the set aggregate object size value, then from the aggregate object collection... Remove the selected aggregate object, and record the name, allocated size and reference count of the removed aggregate object to the metadata management node, then return to step S433 for execution.

[0035] Furthermore, in the above method, after step S6, which attempts to write the request file to the hard disk of the corresponding storage node, the method further includes:

[0036] Step S7: If the request to write to the file fails or the file is deleted, a request to reduce the reference count is sent to the aggregate object generator node.

[0037] Furthermore, in the above method, step S7, sending a request to reduce the reference count to the aggregate object generator node, includes:

[0038] Step S71: Based on the write name of the aggregate object in the request to reduce the reference count of the aggregate object, find the IP address of the aggregate object generator node from which the aggregate object name comes; based on the IP address of the aggregate object generator node, send a request to reduce the reference count to the corresponding aggregate object generator node. The request to reduce the reference count of the aggregate object includes: reducing the write name of the aggregate object.

[0039] Step S72: Based on the generator node sending a request to reduce the reference count, search for the write name of the reduced aggregate object in the metadata management node. If the metadata management node contains the write name of the reduced aggregate object, decrement the reference count of the reduced aggregate object by 1 in the metadata management node; otherwise, in the aggregate object set... In the search, reduce the write name of the aggregate object, if the aggregate object collection If a write name for a reduced aggregate object exists, then the reference count of the reduced aggregate object is decremented by 1; if the aggregate object collection... If the write name of the reduced aggregate object does not exist, then failure is returned;

[0040] Step S73, if the metadata management node or aggregate object set If the reference count of an aggregate object in the metadata management node is zero, then the aggregate object with the zero reference count is removed from the metadata management node or the aggregate object collection. Delete it.

[0041] Furthermore, in the above method, after recording the file ID, requested file size, write name of the new aggregate object, and write offset of the new aggregate object to the metadata management node in step S6, it also includes:

[0042] Step S81: Obtain a read request, and based on the read request, obtain the file ID, write name and write offset of the aggregate object, and file size from the metadata management node;

[0043] In step S82, the location generator calculates the path of the aggregate object in the local file system based on the write name of the aggregate object, and then reads data from the local file system based on the local file system path, the write offset of the aggregate object, and the file size.

[0044] According to another aspect of the invention, a computer-readable medium is provided having computer-readable instructions stored thereon, which can be executed by a processor to implement the method of any one of claims 1 to 8.

[0045] According to another aspect of the present invention, an apparatus for information processing at a network device is provided, the apparatus comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the apparatus is triggered to execute the method of any one of claims 1 to 8.

[0046] Compared to existing technologies, the real-time IO aggregation in this invention can be used in full write IO scenarios. A full write refers to a write operation starting from zero offset, where the write size equals the actual size of the file. The aggregation object in this invention has attributes such as {write name + allocated size + reference count}, and the final aggregated object file size does not exceed a set size.

[0047] This invention includes: generating aggregate objects using an aggregate object generator node device; writing I / O into the newly generated aggregate object; and all subsequent reads of this I / O will access the aggregate object. The aggregate object has attributes such as {write name + allocated size + reference count}, and the final aggregate object file size does not exceed a set size. Before writing I / O, the metadata management node is queried. For new file writes, a request to generate an aggregate object is sent to the aggregate object generator node to obtain the write name and write offset of the aggregate object. For overwriting I / O of existing files, if the write request does not exceed the original size, the original aggregate object is used for writing; for the excess portion, a remaining size request is sent to the aggregate object generator node to obtain a new aggregate object. The real-time I / O aggregation method provided by this invention can work efficiently in distributed storage systems without distributed locks and greatly reduces the total number of files ultimately written to the disk. In practice, performance does not degrade when writing over 10 billion files.

[0048] In this invention, metadata and data are separated. The metadata management node is responsible for managing the metadata of the entire system, while the data portion is written to the local file system using a hashing method. In existing solutions, when the number of files reaches a certain scale, the write speed of the local file system gradually decreases as the number of files continues to increase. In scenarios involving massive file data writing, the system's write performance will be significantly affected, severely impacting the user experience. The real-time IO aggregation method for distributed storage systems provided by this invention can solve the problem of performance degradation when writing massive amounts of file data, enabling the writing of over 10 billion files without performance degradation.

[0049] The file aggregation generator provided by the method of this invention does not use distributed locks, and all generators are completely equal, which greatly reduces complexity while ensuring performance and provides a solid foundation for real-time IO aggregation. Attached Figure Description

[0050] Figure 1 is a schematic diagram of a storage topology according to an embodiment of the present invention;

[0051] Figure 2 is a file reading I / O flowchart according to an embodiment of the present invention;

[0052] Figure 3 is a flowchart of the aggregation object generator node processing the application for generating a new aggregation object according to an embodiment of the present invention;

[0053] Figure 4 is a flowchart of the aggregate object generator node processing aggregate object reference count reduction application according to an embodiment of the present invention;

[0054] Figure 5 is a flowchart of the complete file writing I / O process according to an embodiment of the present invention. Detailed Implementation

[0055] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0056] The topology of the storage system designed in this invention is shown in Figure 1. The storage system may include: a storage network, metadata management nodes, aggregate object generator nodes, containers and object collections, file location generators, storage nodes, hard disks on the storage nodes, and local file systems formatted on the hard disks.

[0057] As shown in Figure 2, the present invention provides a method for real-time I / O aggregation in a distributed storage system, comprising:

[0058] Step S1: Obtain the complete write IO request for writing to a file. The complete write IO request includes: file path, offset, and requested file size, wherein the offset is 0.

[0059] Here, a full write refers to a write operation starting from offset zero, with the write size equal to the actual size of the file. An application can send a full write I / O request to the storage system with the file path, offset 0, and requested file size.

[0060] Step S2: Based on the complete write IO request, query the metadata management node to see if the requested write file exists;

[0061] Here, after receiving a write request, the storage system first checks whether the corresponding file exists in the metadata management node; if the file does not exist, it performs a new file write; if the file exists, it obtains the file ID, the original aggregate object write offset, and the requested file size, and then performs an overwrite write.

[0062] Before writing a file completely (starting from offset zero, the write size equals the actual file size), the metadata management node is queried to determine if the file to be written already exists. The processing methods for writing a new file and overwriting are different:

[0063] If the corresponding file to be written does not exist, i.e., write a new file: first, pre-allocate a file ID from the metadata management node, send a request containing the file ID and file write size information to the aggregate object generator node, and obtain the write name of the aggregate object and the offset of this request in the aggregate object from the aggregate object generator node;

[0064] If the corresponding file already exists (i.e., an overwrite scenario), obtain the file ID, original aggregate object write name, write position, and file size from the metadata management node: 1) If the current write size is less than or equal to the original file size, continue using the original aggregate object write name and write position; 2) If the current write size is greater than the original file size, the portion less than or equal to the original file size continues to use the original aggregate object write name and write position, while the portion exceeding the original file size sends a request containing the remaining size and file ID information to the aggregate object generator node to generate a new aggregate object write name and write position.

[0065] Both new writes and overwrites will obtain information such as file ID, aggregate object write name, write position, and write size. According to the aggregate object write name, the file position generator locates the hard disk to be written to this time, and writes the data to the local file system according to the write position (offset). After successful writing, the file ID, file size, aggregate object write name and write position are recorded in the metadata management server, as shown in Figure 2.

[0066] Step S3: If the file does not exist, obtain the write name and write offset of the new aggregate object corresponding to the file to be written.

[0067] Here, if the file does not exist, a new file will be written.

[0068] Step S4: If the file exists, obtain the write name and write offset of the new aggregate object corresponding to the remaining size of the file to be written, where the remaining size of the file to be written = the size of the file to be written - the size of the original file;

[0069] Here, if the file exists, the file ID, the original aggregate object's write offset, and the file size can be obtained, and an overwrite can be performed subsequently;

[0070] Step S5: Using the location generator, generate the actual write path to the local file system based on the write name of the new aggregate object obtained in step S3; based on the actual write path to the local file system, the file size, and the write offset of the new aggregate object obtained in step S3, attempt to write the requested file to the hard disk of the corresponding storage node; or, using the location generator, generate the corresponding actual write paths to the local file system based on the write names of the original aggregate object and the new aggregate object obtained in step S4; based on the actual write path to the local file system, the size of the requested file, and the write offset of the new aggregate object obtained in step S4, attempt to write the requested file to the hard disk of the corresponding storage node.

[0071] Here, the write name and write offset of the new aggregate object obtained in step S3 or step S4 can be used to generate the actual write path to the local file system through the location generator, and the write offset and the corresponding write size can be combined to complete this write.

[0072] Step S6: If the file write request is successful, record the file ID, the file size requested, the write name of the new aggregate object, and the write offset of the new aggregate object in the metadata management node.

[0073] If the write operation is successful, the file ID, file size, write name of the new aggregate object, and write offset of the new aggregate object will be recorded in the metadata management node.

[0074] In one embodiment of the method for real-time I / O aggregation in a distributed storage system according to the present invention, step S3, if the file does not exist, involves obtaining the write name and write offset of the new aggregation object corresponding to the requested write to the file, including:

[0075] Step S31: If the file does not exist, the aggregation object generator node sends a request to generate a new aggregation object corresponding to the file to be written, and obtains from the aggregation object generator node the write name and write offset of the new aggregation object corresponding to the file to be written, based on the request to generate the new aggregation object corresponding to the file to be written; wherein, the request to generate the new aggregation object corresponding to the file to be written includes: the file ID to be written and the file size;

[0076] Here, for writing a new file: a request to generate a new aggregate object corresponding to the file to be written is sent to the aggregate object generator node. The request to generate a new aggregate object corresponding to the file to be written includes: file ID and write size; and the write name and write offset of the generated new aggregate object are obtained from the response of the aggregate object generator node.

[0077] In one embodiment of the method for real-time I / O aggregation in a distributed storage system according to the present invention, step S31, obtaining from the aggregation object generator node a request to generate a new aggregation object corresponding to the requested write file, and the write name and write offset of the new aggregation object corresponding to the requested write file, are returned:

[0078] Step S311: The aggregate object generator node loads the aggregate object collection from the memory file system. Among them, the aggregate object collection The properties of each aggregate object in the document include: write name, allocated size, and reference count;

[0079] Here, the final file size of the aggregated object does not exceed the preset upper limit of the aggregated object size;

[0080] Step S312: The aggregation object generator node reads the configuration file information, which includes: the preset upper limit of the aggregation object size, the preset size N of the aggregation object set, the local backend IP address, and the network card MAC address and host name corresponding to the local backend IP address.

[0081] Step S313, if the aggregated object collection If the total number M of aggregated objects in the set is less than the preset size N, then in the set of aggregated objects Continue creating NM aggregate objects, and configure the write name, allocated size and reference count for each newly created aggregate object. The write name of each newly created aggregate object is "HASH(MAC+HostName)" + "128-bit random integer value", the initial value of the allocated size is set to zero, and the initial value of the reference count is set to zero.

[0082] Here, the aggregate object generator node device consists of multiple generators, each located on a different node and completely independent of each other.

[0083] The write name of the aggregate object generated by each aggregate object generator node mainly consists of two parts: "HASH(MAC+HostName)" + "128-bit random integer value". The hash value is calculated by combining the MAC address of the network card where the node IP address is located and the hostname. This ensures that the write names of the aggregate objects generated by each generator node are different, while maintaining the mapping relationship between the calculated hash value and the IP address. This allows the node IP from which the generator came to be found quickly by reverse engineering through the file name. The random value is used to distinguish the write names within the generator.

[0084] Each aggregate object generator node maintains its own aggregate object {write name + allocation size + reference count} set in memory. A certain number of write names are generated in advance. For each write name generated, the storage location is calculated by the file location generator and a file with the same name is created. Each write name corresponds to a pre-allocated write size and a reference count. The initial size and reference count are both zero.

[0085] Step S314, from the aggregate object collection Select the aggregate object with the smallest allocated size.

[0086] Step S315: If the allocated size of the selected aggregation object plus the requested write file size is less than or equal to the set aggregation object size value, then the aggregation object is selected;

[0087] Step S316: Update the allocated size and reference count of the selected aggregate object, where the allocated size = the original allocated size of the selected aggregate object + the requested write size, and the reference count = the original reference count of the selected aggregate object + 1.

[0088] Step S317: The original allocated size of the selected aggregate object is used as the write offset of the selected aggregate object. The aggregate object generator node sends a request to generate a new aggregate object corresponding to the file to be written, and feeds back the write offset and write name of the selected aggregate object corresponding to the file to be written.

[0089] Here, the write offset and write name of the selected aggregate object can be fed back to the aggregate object generator node; upon receiving the feedback, the aggregate object generator node retrieves the aggregate object write name and write offset from the feedback.

[0090] Step S318: If the allocated size of the selected aggregate object plus the requested write file size is greater than the set aggregate object size value, then retrieve the aggregate object from the aggregate object collection. Remove the selected aggregate object, and record the name, allocated size and reference count of the removed aggregate object to the metadata management node. Then, return to step S313 to continue creating new aggregate objects for selection.

[0091] Here, when the aggregate object generator receives a request to obtain an aggregate object containing the write request size, it selects the write name with the smallest allocation size from the set {write name + allocation size + reference count}. If "minimum allocation size + requested write size <= set aggregate object size", then that object is selected.

[0092] Otherwise, remove the record from the collection, record the aggregate object in the metadata collection including {write name + allocation size + reference count}, and create a new aggregate object and a file simultaneously. Then, use the newly generated aggregate object as the selected object. After selecting the aggregate object, add the original allocation size to the requested write size and update the allocation size (space is reserved when requesting the aggregate object for later writing), and increment the reference count by 1 (the reference count is decremented upon deletion; if the reference count is 0, the aggregate object is deleted). Finally, send the aggregate object's write name and the original allocation size (before the update) as the write location to the request sender. At this point, the object generator completes this generation request, as shown in Figure 4.

[0093] In one embodiment of the method for real-time I / O aggregation in a distributed storage system according to the present invention, step S4, if the file exists, obtains the write name and write offset of the new aggregation object corresponding to the remaining size of the requested file to be written, including:

[0094] Step S41: If the file exists, compare the size of the file to be written with the size of the original file.

[0095] Step S42: If the size of the file to be written is less than or equal to the size of the original file, then the aggregate object of the original file is used;

[0096] Step S43: If the size of the requested file to be written is greater than the size of the original file, then the portion of the requested file to be written that is less than or equal to the size of the original file uses the aggregation object of the original file. "The remaining size of the requested file to be written = the size of the requested file to be written - the size of the original file". A request to generate a new aggregation object corresponding to the remaining size of the requested file to be written is sent to the aggregation object generator node. This request includes: the requested file ID and the remaining size of the requested file to be written. The request is then obtained from the aggregation object generator node, and based on this request, the write name and write offset of the new aggregation object corresponding to the remaining size of the requested file to be written are fed back.

[0097] Here, for overwrite writes: The relationship between the size of the requested file to be written and the size of the original file is determined. If the size of the requested file to be written is less than or equal to the size of the original file, the original aggregate object of the original file is used. If the size of the requested file to be written is greater than the size of the original file, the portion of the requested file to be written that is less than or equal to the size of the original file uses the original aggregate object of the original file. "The remaining size of the requested file to be written = new write size - original file size" is defined by sending a request to the aggregate object generator node to generate a new aggregate object corresponding to the remaining size of the requested file to be written. This request includes: the ID of the requested file to be written, the remaining new write size, and the request to generate a new aggregate object based on the remaining size of the requested file to be written is obtained from the aggregate object generator node. The response includes the write name and write offset of the new aggregate object corresponding to the remaining size of the requested file to be written.

[0098] In one embodiment of the method for real-time I / O aggregation in a distributed storage system according to the present invention, step S43, obtaining from the aggregation object generator node, the write name and write offset of the new aggregation object corresponding to the remaining size of the requested file to be written, based on the request to generate the new aggregation object, includes:

[0099] Step S431: The aggregate object generator node loads the aggregate object collection from the memory file system. Among them, the aggregate object collection The properties of each aggregate object in the document include: write name, allocated size, and reference count;

[0100] Here, the final file size of the aggregated object does not exceed the preset upper limit of the aggregated object size;

[0101] Step S432: The aggregation object generator node reads the configuration file information, which includes: the preset upper limit of the aggregation object size, the preset size N of the aggregation object set, the local backend IP address, and the network card MAC address and host name corresponding to the local backend IP address.

[0102] Step S433, if the aggregated object collection If the total number M of aggregated objects in the set is less than the preset size N, then in the set of aggregated objects Continue creating NM aggregate objects, and configure the write name, allocated size and reference count for each newly created aggregate object. The write name of each newly created aggregate object is "HASH(MAC+HostName)" + "128-bit random integer value", the initial value of the allocated size is set to zero, and the initial value of the reference count is set to zero.

[0103] Step S434, from the aggregate object collection Select the aggregate object with the smallest allocated size.

[0104] Step S435: If the allocated size of the selected aggregation object plus the remaining size to be written to the file is less than or equal to the set size value of the aggregation object, then the aggregation object is selected;

[0105] Step S436: Update the allocated size and reference count of the selected aggregate object, where the allocated size = the original allocated size of the selected aggregate object + the remaining size to be written to the file, and the reference count = the original reference count of the selected aggregate object + 1.

[0106] Step S437: The original allocated size of the selected aggregate object is used as the write offset of the selected aggregate object. The aggregate object generator node sends a request to generate a new aggregate object corresponding to the file to be written, and feeds back the write offset and write name of the selected aggregate object corresponding to the file to be written.

[0107] Here, the write offset and write name of the selected aggregate object can be fed back to the aggregate object generator node; upon receiving the feedback, the aggregate object generator node retrieves the aggregate object write name and write offset from the feedback.

[0108] Step S438: If the allocated size of the selected aggregate object plus the remaining size to be written to the file is greater than the set aggregate object size value, then retrieve the aggregate object from the aggregate object collection. Remove the selected aggregate object, and record the name, allocated size and reference count of the removed aggregate object to the metadata management node. Then, return to step S433 to continue creating new aggregate objects for selection.

[0109] In one embodiment of the method for real-time I / O aggregation in a distributed storage system according to the present invention, after step S6, which attempts to write the requested file to the hard disk of the corresponding storage node, the method further includes:

[0110] Step S7: If the request to write to the file fails or the file is deleted, a request to reduce the reference count is sent to the aggregate object generator node.

[0111] If the write operation fails, a request to reduce the reference count is sent to the aggregate object generator node to roll back the reference count.

[0112] In one embodiment of the method for real-time I / O aggregation in a distributed storage system according to the present invention, step S7, sending a request to reduce the reference count to the aggregation object generator node, includes:

[0113] Step S71: Based on the write name of the aggregate object in the request to reduce the reference count of the aggregate object, find the IP address of the aggregate object generator node from which the aggregate object name comes; based on the IP address of the aggregate object generator node, send a request to reduce the reference count to the corresponding aggregate object generator node. The request to reduce the reference count of the aggregate object includes: reducing the write name of the aggregate object.

[0114] Here, the client can look up the IP address of the aggregate object generator node from which the aggregate object name comes based on the write name of the aggregate object in the request to reduce the reference count of the aggregate object, and send a request to reduce the reference count to the corresponding aggregate object generator node based on the IP address of the aggregate object generator node.

[0115] The aggregate object generator node receives a request to reduce the reference count. The request includes: the aggregate object name, and the reference count reduction value (-1).

[0116] Step S72: Based on the generator node sending a request to reduce the reference count, search for the write name of the reduced aggregate object in the metadata management node. If the metadata management node contains the write name of the reduced aggregate object, decrement the reference count of the reduced aggregate object by 1 in the metadata management node; otherwise, in the aggregate object set... In the search, reduce the write name of the aggregate object, if the aggregate object collection If a write name for a reduced aggregate object exists, then the reference count of the reduced aggregate object is decremented by 1; if the aggregate object collection... If the write name of the reduced aggregate object does not exist, then failure is returned;

[0117] Step S73, if the metadata management node or aggregate object set If the reference count of an aggregate object in the metadata management node is zero, then the aggregate object with the zero reference count is removed from the metadata management node or the aggregate object collection. Delete it.

[0118] Here, if the reference count drops to zero, the corresponding aggregate object needs to be deleted.

[0119] The aggregate object generator node also provides an interface for reducing the reference count and obtaining the value of the reference count before the reduction. In scenarios such as when a file is deleted, or when a file sends a request to the aggregate object generator node and receives a response but the write operation fails, it is necessary to reduce the reference count of the aggregate object. When the reference count of the aggregate object is about to reach zero, the corresponding aggregate object needs to be deleted, as shown in Figure 4.

[0120] As shown in Figure 5, in one embodiment of the method for real-time I / O aggregation in a distributed storage system according to the present invention, after recording the file ID, the requested file size, the write name of the new aggregation object, and the write offset of the new aggregation object to the metadata management node in step S6, the method further includes:

[0121] Step S81: Obtain a read request, and based on the read request, obtain the file ID, write name and write offset of the aggregate object, and file size from the metadata management node;

[0122] Here, the application sends a read request to the storage system; after receiving the read request, the storage system obtains the file ID, the write name and write offset of the aggregate object, and the file size from the metadata management node;

[0123] In step S82, the location generator calculates the path of the aggregate object in the local file system based on the write name of the aggregate object, and then reads data from the local file system based on the local file system path, the write offset of the aggregate object, and the file size.

[0124] Here, when reading a file, you can first obtain the write name, write offset, and file size of the aggregate object recorded in the metadata management node, and then obtain the specific location of the aggregate object in the file system through the file location generator, and read the corresponding data from the local file system to complete the reading, as shown in Figure 5.

[0125] It can read the collection of aggregate objects in a loop until all aggregate objects have been read. All reads are strictly in the order they are read and the data is put into the memory data buffer in sequence.

[0126] In summary, the real-time IO aggregation in the method of this invention can be used in full write IO scenarios. A full write refers to a write operation starting from zero offset, where the write size equals the actual size of the file. The aggregation object in the method of this invention has attributes such as {write name + allocated size + reference count}, and the final size of the aggregation object file does not exceed the set size.

[0127] This invention includes: generating aggregate objects using an aggregate object generator node device; writing I / O into the newly generated aggregate object; and all subsequent reads of this I / O will access the aggregate object. The aggregate object has attributes such as {write name + allocated size + reference count}, and the final aggregate object file size does not exceed a set size. Before writing I / O, the metadata management node is queried. For new file writes, a request to generate an aggregate object is sent to the aggregate object generator node to obtain the write name and write offset of the aggregate object. For overwriting I / O of existing files, if the write request does not exceed the original size, the original aggregate object is used for writing; for the excess portion, a remaining size request is sent to the aggregate object generator node to obtain a new aggregate object. The real-time I / O aggregation method provided by this invention can work efficiently in distributed storage systems without distributed locks and greatly reduces the total number of files ultimately written to the disk. In practice, performance does not degrade when writing over 10 billion files.

[0128] In this invention, metadata and data are separated. The metadata management node is responsible for managing the metadata of the entire system, while the data portion is written to the local file system using a hashing method. In existing solutions, when the number of files reaches a certain scale, the write speed of the local file system gradually decreases as the number of files continues to increase. In scenarios involving massive file data writing, the system's write performance will be significantly affected, severely impacting the user experience. The real-time IO aggregation method for distributed storage systems provided by this invention can solve the problem of performance degradation when writing massive amounts of file data, enabling the writing of over 10 billion files without performance degradation.

[0129] The file aggregation generator provided by the method of this invention does not use distributed locks, and all generators are completely equal, which greatly reduces complexity while ensuring performance and provides a solid foundation for real-time IO aggregation.

[0130] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0131] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0132] Obviously, those skilled in the art can make various modifications and variations to the invention without departing from the spirit and scope of the invention. Therefore, if these modifications and variations fall within the scope of the claims of the invention and their equivalents, the invention is also intended to include these modifications and variations.

Claims

1. A method for real-time I / O aggregation in a distributed storage system, characterized in that, include: Step S1: Obtain the complete write IO request for the requested file, which includes: file path, offset, and requested file size, wherein the offset is 0; Step S2: Based on the complete write IO request, query the metadata management node to see if the requested file exists; Step S3: If the file does not exist, obtain the write name and write offset of the new aggregate object corresponding to the requested file; Step S4: If the file exists, obtain the write name and write offset of the new aggregate object corresponding to the remaining size of the requested file, wherein the remaining size of the requested file = requested file size - original file size; Step S5: Based on the write name of the new aggregate object obtained in step S3, the location generator generates the actual write path to the local file system; based on the actual write path to the local file system, the file size, and the write offset of the new aggregate object obtained in step S3, it attempts to write the requested file to the hard disk of the corresponding storage node; or, based on the write name of the original aggregate object and the new aggregate object obtained in step S4, the location generator generates the corresponding actual write path to the local file system; based on the actual write path to the local file system, the size of the requested file, and the write offset of the new aggregate object obtained in step S4, it attempts to write the requested file to the hard disk of the corresponding storage node. Step S6: If the requested file write is successful, record the file ID, requested file size, new aggregate object write name, and new aggregate object write offset to the metadata management node; Step S4: If the file exists, obtain the write name and write offset of the new aggregate object corresponding to the remaining size of the requested file, including: Step S41: If the file exists, compare the size of the requested file with the size of the original file; Step S42: If the size of the requested file is <= the size of the original file, use the aggregate object of the original file; Step S43: If the size of the requested file is > the size of the original file, use the portion of the requested file less than or equal to the size of the original file. Using the original file's aggregate object; requesting the remaining size of the file to be written = requesting the size of the file to be written - the size of the original file; sending a request to the aggregate object generator node to generate a new aggregate object corresponding to the remaining size of the file to be written, wherein the request to generate a new aggregate object corresponding to the remaining size of the file to be written includes: the file ID to be written, and the remaining size of the file to be written; and obtaining from the aggregate object generator node, based on the request to generate a new aggregate object corresponding to the remaining size of the file to be written, feedback the write name and write offset of the new aggregate object corresponding to the remaining size of the file to be written;Step S43: Obtain from the aggregate object generator node the request to generate a new aggregate object corresponding to the remaining size of the requested file to be written, and the feedback includes the write name and write offset of the new aggregate object corresponding to the remaining size of the requested file to be written, including: Step S431: The aggregate object generator node loads the aggregate object set Ø from the memory file system, wherein the attributes of each aggregate object in the aggregate object set Ø include: write name, allocated size, and reference count; Step S432: The aggregate object generator node reads the configuration file information, wherein the configuration file information includes: a preset upper limit value of the aggregate object size, a preset size N of the aggregate object set, the local backend IP address, and the MAC address and host name of the network card corresponding to the local backend IP address; Step S433: If the total number M of aggregate objects in the aggregate object set Ø is less than the preset size N, then NM aggregate objects are created in the aggregate object set Ø, and the write name, allocated size, and reference count are configured for each newly created aggregate object, wherein the write name of each newly created aggregate object = "HASH(MAC+HostName)" + "128-bit random integer value", the initial value of the allocated size is set to zero, and the initial value of the reference count is set to zero; Step S434, select the aggregate object with the smallest allocated size from the aggregate object set Ø; Step S435, if the allocated size of the selected aggregate object + the remaining size to be written to the file is <= the set aggregate object size value, then the aggregate object is selected; Step S436, update the allocated size and reference count of the selected aggregate object, where the allocated size = the original allocated size of the selected aggregate object + the remaining size to be written to the file, and the reference count = Step S437: Increment the original reference count of the selected aggregate object by 1; Step S438: Use the original allocated size of the selected aggregate object as the write offset of the selected aggregate object. The aggregate object generator node sends a request to generate a new aggregate object corresponding to the file to be written, and feeds back the write offset and write name of the selected aggregate object corresponding to the file to be written; Step S439: If the allocated size of the selected aggregate object + the remaining size of the file to be written > the set aggregate object size value, remove the selected aggregate object from the aggregate object set Ø, and record the write name, allocated size and reference count of the removed aggregate object to the metadata management node, and then return to step S433 for execution; Step S6: After attempting to write the file to be written to the hard disk of the corresponding storage node, the process also includes: Step S7: If the file writing fails or the file is deleted, send a request to reduce the reference count to the aggregate object generator node.Step S7, sending a request to reduce the reference count to the aggregate object generator node, includes: Step S71, finding the IP address of the aggregate object generator node from which the aggregate object name originates based on the write name of the aggregate object in the request to reduce the reference count of the aggregate object, and sending a request to reduce the reference count to the corresponding aggregate object generator node based on the IP address of the aggregate object generator node, wherein the request to reduce the reference count of the aggregate object includes: reducing the write name of the aggregate object; Step S72, based on the request to reduce the reference count sent by the generator node, searching for the write name of the aggregate object to be reduced in the metadata management node. If the metadata management node contains a write name for a reduced aggregate object, then the reference count of the reduced aggregate object is decremented by 1 in the metadata management node; otherwise, the write name for the reduced aggregate object is searched in the aggregate object set Ø. If the write name for the reduced aggregate object exists in the aggregate object set Ø, then the reference count of the reduced aggregate object is decremented by 1; if the write name for the reduced aggregate object does not exist in the aggregate object set Ø, then failure is returned; Step S73: If the reference count of an aggregate object in the metadata management node or the aggregate object set Ø is zero, then the aggregate object with a reference count of zero is deleted from the metadata management node or the aggregate object set Ø.

2. The method for real-time I / O aggregation in a distributed storage system as described in claim 1, characterized in that, Step S3, if the file does not exist, obtain the write name and write offset of the new aggregate object corresponding to the file to be written, including: Step S31, if the file does not exist, the aggregate object generator node sends a request to generate a new aggregate object corresponding to the file to be written, and obtains the write name and write offset of the new aggregate object corresponding to the file to be written from the aggregate object generator node based on the request to generate the new aggregate object corresponding to the file to be written, wherein the request to generate the new aggregate object corresponding to the file to be written includes: the file ID to be written and the file size.

3. The method for real-time I / O aggregation in a distributed storage system as described in claim 2, characterized in that, Step S31: Obtain the request for a new aggregate object corresponding to the application to write the file from the aggregate object generator node, and receive the write name and write offset of the new aggregate object corresponding to the application to write the file. This includes: Step S311: The aggregate object generator node loads the aggregate object set Ø from the memory file system, wherein the attributes of each aggregate object in the aggregate object set Ø include: write name, allocated size, and reference count; Step S312: The aggregate object generator node reads the configuration file information, which includes: a preset upper limit for the aggregate object size, a preset size N for the aggregate object set, the local backend IP address, and the MAC address and host name of the network card corresponding to the local backend IP address; Step S313: If the total number M of aggregate objects in the aggregate object set Ø is less than the preset size N, then NM aggregate objects are created in the aggregate object set Ø, and a write name, allocated size, and reference count are configured for each newly created aggregate object, wherein the write name of each newly created aggregate object = "HASH(MAC+HostName)" + "128-bit random integer value", the initial value of the allocated size is set to zero, and the initial value of the reference count is set to zero; Step S314, select the aggregate object with the smallest allocated size from the aggregate object set Ø; Step S315, if the allocated size of the selected aggregate object + the requested write file size <= the set aggregate object size value, then select the aggregate object; Step S316, update the allocated size and reference count of the selected aggregate object, where the allocated size = the original allocated size of the selected aggregate object + the requested write size, and the reference count = The original reference count of the selected aggregate object is incremented by 1; in step S317, the original allocated size of the selected aggregate object is used as the write offset of the selected aggregate object, and the aggregate object generator node sends a request to generate a new aggregate object corresponding to the file to be written, and feeds back the write offset and write name of the selected aggregate object corresponding to the file to be written; in step S318, if the allocated size of the selected aggregate object + the size of the file to be written > the set aggregate object size value, the selected aggregate object is removed from the aggregate object set Ø, and the write name, allocated size and reference count of the removed aggregate object are recorded to the metadata management node, and then the process returns to step S313.

4. The method for real-time I / O aggregation in a distributed storage system as described in claim 1, characterized in that, Step S6, after recording the file ID, requested file size, new aggregate object's write name, and new aggregate object's write offset to the metadata management node, further includes: Step S81, obtaining a read request, and obtaining the file ID, aggregate object's write name, write offset, and file size from the metadata management node based on the read request; Step S82, the location generator calculates the local file system path of the aggregate object based on its write name, and then reads data from the local file system based on the local file system path, aggregate object's write offset, and file size.

5. A computer-readable medium having stored thereon computer-readable instructions that can be executed by a processor to implement the method of any one of claims 1 to 4.

6. An apparatus for information processing at a network device end, the apparatus comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein, When the computer program instructions are executed by the processor, the device is triggered to perform the method of any one of claims 1 to 4.

Images