Dynamic pointer conversion method and device for distributed system

Abstract

The embodiment of the invention provides a pointer dynamic conversion method and device for a distributed system, and relates to the technical field of data processing.The method specifically comprises the steps that a write-in request of target data is received; generating a target record corresponding to the target IO request based on the target data, and inserting the target record into a target node in each copy cache space; the cache space is set to be of a tree structure; a data structure used for storing the target record in the target node adopts a secondary pointer structure; the secondary pointer structure comprises a target record, metadata and a mapping address in the target record, and a target address pointed by the mapping address; when the state of the target record is a submission state or a pause state, adjusting a data structure used for storing the target record in the target node into a first-level pointer structure; the first-level pointer structure comprises a target record corresponding to the target IO request, and metadata and a target address in the target record. The memory utilization rate can be improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of data processing, and in particular to a pointer dynamic conversion method and device for a distributed system. Background Art

[0002] In the current storage technology field, the metadata occupancy rate of the hard disk pool is a key issue. Especially in large-capacity storage scenarios, such as when using HDD hard disks, the space occupied by metadata becomes particularly significant. In traditional metadata management methods, each minimum metadata entry occupies 64 bytes, and based on an average of 4KB per metadata entry, metadata accounts for as much as 15% of the overall storage space. This ratio will lead to serious space waste in large-capacity storage scenarios.

[0003] In particular, in all-flash storage products, in order to ensure that there is enough space to store metadata, more persistent memory needs to be configured. However, compared with HDD hard disks, the capacity of SSD disks is generally smaller, which means that in all-flash scenarios, although the metadata space is relatively sufficient, it faces severe challenges in large-capacity HDD scenarios. For example, in a single-node storage system consisting of 12 10T hard disks, the total capacity is 120T, and the required metadata space is as high as 1800GB (calculated at a ratio of 15 thousandths). However, in actual configuration, two 128GB persistent memories only provide a total of 256GB of space, which is far from meeting the needs of large-capacity HDD scenarios.

[0004] In order to meet the requirements of large-capacity HDD disks and reduce the proportion of metadata in the overall storage space, the size of metadata must be further reduced. Therefore, how to effectively reduce the size of metadata and improve the utilization of storage space has become an important issue that needs to be solved in the current storage technology field. Summary of the invention

[0005] In view of this, an embodiment of the present application provides a pointer dynamic conversion method and device for a distributed system, and the present application can improve memory utilization.

[0006] In a first aspect, an embodiment of the present application provides a pointer dynamic conversion method for a distributed system, which is applied in a distributed system and specifically includes:

[0007] Receive a write request for target data; the target data includes metadata corresponding to the target IO request and a physical address corresponding to the data requested by the target IO request;

[0008] Generate a target record corresponding to the target IO request based on the target data, and insert the target record into a target node in each replica cache space; the cache space is set as a tree structure; the data structure for storing the target record in the target node adopts a two-level pointer structure; the two-level pointer structure includes the target record, the metadata and mapping address in the target record, and the target address pointed to by the mapping address.

[0009] Obtain the status of the target record in each replica.

[0010] When the status of the target record is the committed state or the aborted state, adjust the data structure for storing the target record in the target node to a one-level pointer structure; the one-level pointer structure includes the target record corresponding to the target IO request, the metadata and the target address in the target record.

[0011] As an optional implementation manner of an embodiment of the present application, after generating a target record corresponding to the target IO request based on the target data and inserting the target record into a target node in each replica cache space, the method further includes:

[0012] Obtain the status of the target record in each replica. When the status of the target record is the active state, maintain the two-level pointer structure of the target node.

[0013] As an optional implementation manner of an embodiment of the present application, the obtaining the status of the target record in each replica includes:

[0014] Periodically traverse the records in each node on the tree structure in each replica to obtain the status of the target record in each replica.

[0015] As an optional implementation manner of an embodiment of the present application, before generating an initial record based on the metadata and the target address and inserting the initial record into multiple target nodes, the method further includes:

[0016] Create the tree structure in each replica respectively.

[0017] After receiving a write request for the target data, obtain the target node corresponding to the target record in the tree structure of each replica.

[0018] As an optional implementation manner of an embodiment of the present application, the method further includes:

[0019] Whenever the distributed system powers off and restarts, reconstruct the tree structure in each replica.

[0020] As an optional implementation manner of an embodiment of the present application, before generating a target record corresponding to the target IO request based on the target data and inserting the target record into a target node in each copy cache space, the method further includes:

[0021] Based on the target address, a hash algorithm is used to generate the mapping address, and based on the mapping address, the metadata, and the target address, the target record is generated.

[0022] As an optional implementation manner of an embodiment of the present application, before receiving a write request for target data, the method further includes:

[0023] After receiving a target IO request sent by a client, obtain the metadata and the target address corresponding to the target IO request to generate a write request for the target data.

[0024] In a second aspect, an embodiment of the present application provides a pointer dynamic conversion device for a distributed system, including:

[0025] A receiving unit, configured to receive a write request for target data; the target data includes metadata corresponding to a target IO request and a physical address corresponding to the data requested by the target IO request;

[0026] An inserting unit, configured to generate a target record corresponding to the target IO request based on the target data and insert the target record into a target node in each copy cache space; the cache space is set as a tree structure; a data structure for storing the target record in the target node adopts a two-level pointer structure; the two-level pointer structure includes the target record, the metadata and the mapping address in the target record, and the target address pointed to by the mapping address;

[0027] An obtaining unit, configured to obtain the status of the target record in each copy;

[0028] An adjusting unit, configured to, when the status of the target record is a committed status or an aborted status, adjust the data structure for storing the target record in the target node to a one-level pointer structure; the one-level pointer structure includes the target record corresponding to the target IO request, and the metadata and the target address in the target record.

[0029] As an optional implementation manner of an embodiment of the present application, the adjusting unit is further configured to obtain the status of the target record in each copy, and when the status of the target record is an active status, maintain the two-level pointer structure of the target node.

[0030] As an alternative implementation manner of an embodiment of the present application, the obtaining unit is specifically configured to periodically traverse the records in each node of the tree structure in each copy to obtain the status of the target record in each copy.

[0031] As an alternative implementation manner of an embodiment of the present application, the receiving unit is specifically configured to create the tree structure in each of the copies; after receiving a write request for the target data, obtain the target node corresponding to the target record in the tree structure of each copy.

[0032] As an alternative implementation manner of an embodiment of the present application, the pointer dynamic conversion method device for a distributed system further includes a reply unit, configured to reconstruct the tree structure in each of the copies whenever the distributed system powers off and restarts.

[0033] As an alternative implementation manner of an embodiment of the present application, the inserting unit is specifically configured to generate the mapping address by using a hash algorithm based on the target address, and generate the target record based on the mapping address, the metadata, and the target address.

[0034] As an alternative implementation manner of an embodiment of the present application, the receiving unit is specifically configured to, after receiving a target IO request sent by a client, obtain the metadata and the target address corresponding to the target IO request to generate a write request for the target data.

[0035] In a third aspect, an embodiment of the present application provides an electronic device, including: a memory and a processor, where the memory is used to store a computer program; the processor is configured to, when executing the computer program, enable the electronic device to implement the pointer dynamic conversion method for a distributed system described in any one of the above embodiments.

[0036] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a computing device, the computing device is enabled to implement the pointer dynamic conversion method for a distributed system described in any one of the above embodiments.

[0037] The pointer dynamic conversion method for a distributed system provided by an embodiment of the present application is specifically as follows: receiving a write request for target data; the target data includes metadata corresponding to a target IO request and a physical address corresponding to the data requested by the target IO request; generating a target record corresponding to the target IO request based on the target data, and inserting the target record onto a target node in each replica cache space; the cache space is set as a tree structure; the data structure for storing the target record in the target node adopts a two-level pointer structure; the two-level pointer structure includes the target record, the metadata and mapping address in the target record, and the target address pointed to by the mapping address; obtaining the status of the target record in each replica; when the status of the target record is the committed status or the aborted status, adjusting the data structure for storing the target record in the target node to a one-level pointer structure; the one-level pointer structure includes the target record corresponding to the target IO request, and the metadata and the target address in the target record.

[0038] Through the above method, in the target data writing stage, the present application uses a two-level pointer structure to store the record corresponding to the target data, and thus can flexibly update the mapping address through the two-level pointer structure, obtain the target address through the mapping address, avoid a large number of modifications to the target address corresponding to the record during data migration, and improve the data maintenance efficiency. When the target record is in the committed or aborted state, it is converted to a one-level pointer structure, and the metadata and target address corresponding to the target record are directly obtained through the one-level pointer structure, reducing the addressing steps, reducing the access latency, accelerating the data access speed, and optimizing the overall data processing performance of the system.

[0039] Meanwhile, a tree structure is adopted to store the records. Compared with array storage, nodes can reuse the storage space, reduce the space waste caused by scattered data storage, and improve the utilization rate of storage resources. In addition, by dynamically adjusting the pointer structure, after the two-level pointer structure is adjusted to a one-level pointer structure, the one-level pointer structure is more compact than the two-level pointer structure, reducing unnecessary memory occupation and improving the memory usage efficiency of the system. Description of the Drawings

[0040] The drawings here are incorporated into the specification and form a part of this specification, showing embodiments consistent with the present application and used together with the specification to explain the principles of the present application.

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required to be referred to in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0042] Figure 1 One of the flowchart of steps of the pointer dynamic conversion method for a distributed system provided by an embodiment of the present application;

[0043] Figure 2 Schematic diagram of a secondary pointer structure in the pointer dynamic conversion method for a distributed system provided by an embodiment of the present application;

[0044] Figure 3 Schematic diagram of a primary pointer structure in the pointer dynamic conversion method for a distributed system provided by an embodiment of the present application;

[0045] Figure 4 Another flowchart of steps of the pointer dynamic conversion method for a distributed system provided by an embodiment of the present application;

[0046] Figure 5 Schematic diagram of the structure of a pointer dynamic conversion device for a distributed system provided by an embodiment of the present application;

[0047] Figure 6 Schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0048] In order to more clearly understand the above objects, features and advantages of the present disclosure, the solutions of the present disclosure will be further described below. It should be noted that, without conflict, the embodiments of the present disclosure and the features in the embodiments may be combined with each other.

[0049] Many specific details are set forth in the following description in order to fully understand the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only a part of the embodiments of the present disclosure, rather than all of the embodiments.

[0050] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the embodiments of the present application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Rather, the use of the words "exemplary" or "for example" is intended to present relevant concepts in a specific manner. In addition, in the description of the embodiments of the present application, unless otherwise specified, the meaning of "a plurality" refers to two or more.

[0051] It should be noted that in this article, the term "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article, or device including a series of elements not only includes those elements but also includes other elements not explicitly listed, or further includes elements inherent to such a process, method, article, or device. Without more limitations, an element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, method, article, or device including the said element.

[0052] An embodiment of this application provides a method for dynamically converting pointers for a distributed system. Referring to Figure 1 as shown, the method for dynamically converting pointers for the distributed system includes the following steps S101 - S104:

[0053] S101. Receive a write request for target data.

[0054] Among them, the target data includes metadata corresponding to a target IO request and the physical address corresponding to the data requested by the target IO request.

[0055] In a file system, an IO (Input / Output) request refers to an instruction for a client to perform data input or output operations on a file or storage device; when a client needs to obtain data from the file system, a read request will be initiated. For example, when a user opens a text file to view its content, a text editing software will send a read request to the file system. The file system locates the corresponding position on the storage device (such as a hard disk, solid-state drive) storing the file data according to the request, then reads the data into the memory, and then passes it to the application program for display to the user. The read request may also be used to read metadata information of the file, such as the size and modification time of the file, to update the display of file attributes in the application program. When a client wants to save data to the file system, a write request will be initiated. For example, when a user enters new content in a document editing software and saves it, the editing software will send a write request to the file system. The file system determines the location where the data will be stored (possibly needs to allocate a new storage block), and then writes the data in the memory to the storage device.

[0056] Specifically, when the client needs to access the data at the target address, it sends an IO request to the server or storage device. This request carries various information, such as the amount of data to be read or written, the type of operation (read or write), and the logical location of the target data. To facilitate the processing and scheduling of a large number of IO requests, while ensuring data management, security, monitoring, etc., when the distributed system receives each IO request, it generates the metadata corresponding to the IO request. Among them, the metadata contains various descriptive information of the current IO request, such as the request timestamp, request type, the size of the data requested by the request, the data offset, etc. By using the above multiple data to generate the metadata corresponding to the IO request, and then writing the metadata corresponding to each IO request into the distributed system, it is convenient for the distributed system to obtain rich information about the data through the metadata corresponding to each IO request, improving subsequent data operations, performance optimization, and interoperability between systems.

[0057] It should be noted that the data requested by the target IO request in this application is all stored in the hard disk. Therefore, in the target IO request, it will also include the physical address where the data involved in the request itself is located, that is, the target address of the data on the hard disk. Through this target address, the system can accurately find the physical location of the data in the hard disk, thereby realizing the reading or writing of the data. For example, in a database system, a certain data record may be stored on a specific combination of sectors on the hard disk, and the target address will point to these sectors. When there is an IO request to read this record, the system reads the data from the hard disk according to the target address.

[0058] For example, when the received IO request is a request to read File A, the file system generates the corresponding metadata according to the IO request. Then the current metadata can include information such as the name, size, creation time, modification time, file type of the file. It should be noted that the metadata can also include the target address of File A. However, in the embodiment of this application, the target address of File A is taken out separately from the metadata, and the mapping address in the secondary pointer structure is used to point to the target address later, in order to dynamically update and maintain all records in the node storing the target data, avoiding the problem of a large number of record content modifications due to data migration.

[0059] S102. Generate a target record corresponding to the target IO request based on the target data, and insert the target record into the target node in each replica cache space.

[0060] Among them, the cache space is set as a tree structure; the data structure for storing the target record in the target node adopts a two-level pointer structure; the two-level pointer structure includes the target record, the metadata and mapping address in the target record, and the target address pointed to by the mapping address.

[0061] Specifically, the target record can be understood as a data structure containing important information of the IO request, used to store the important information of the IO request, and then insert the target record corresponding to the target IO request into the target node in the cache space of each replica in the distributed system, so as to maintain and store the target record corresponding to the target IO request through a tree structure.

[0062] It should be noted that in a distributed system, the cache space improves data access efficiency by retaining records corresponding to a limited number of recently received IO requests and eliminating the least recently used records after reaching the upper limit. In the embodiment of the present application, the cache space is set as a tree structure. For example, it can be set as a B+ tree structure, which is a self-balancing multi-way search tree commonly used in databases and file systems because it can effectively support search, insertion, and deletion operations while maintaining the balance of the tree and ensuring that the time complexity of the operations is within...

[0063] Specifically, through the leaf nodes in the B+ tree structure, the leaf nodes contain Key values (key values) corresponding to multiple records. Through the Key values, the two-level pointer structure pointing to the metadata and target address corresponding to each record can be obtained. The multiple records included in the leaf nodes will form a record linked list, which is convenient for sequential access. By traversing the linked list of the leaf nodes, the target records corresponding to all IO requests on the linked list can be quickly obtained.

[0064] Specifically, the two-level pointer structure can be understood as managing the storage of data and the association relationship between data in two layers. The first-level pointer can point to the mapping address, and the second-level pointer is to point to the target address through the mapping address; since in a distributed system, data may be migrated due to reasons such as disk space management and load balancing, by adopting a two-level pointer structure, when deleting a record in the record linked list of a certain node, only the mapping address of other records needs to be modified to make the mapping address of this record continue to point to the original target address, without the need to change the target address corresponding to the record. Without changing the target address of the record itself, the target address pointed to by the mapping address can be flexibly updated, making each record relatively independent to a certain extent and not being affected by the deletion of one record in terms of the storage and pointers of other records, greatly improving the efficiency of data maintenance and avoiding the problem of needing to modify a large amount of record content due to data migration.

[0065] Refer to Figure 2 As shown, it is a schematic diagram of the target record chain corresponding to a certain node in the B+ tree structure. Among them, it includes the record linked list in node 1 of the tree structure. There are n records in this linked list. Through the KEY value corresponding to this record, the metadata and mapping address corresponding to the current record are obtained, and then the target address is obtained through the mapping address.

[0066] In some embodiments, the specific implementation manner of generating the initial record based on the metadata and the target address in the above step S102 may be:

[0067] Based on the target address, a hash algorithm is used to generate the mapping address, and based on the mapping address, the metadata, and the target address, the target record is generated.

[0068] In some embodiments, based on the mapping method of the hash table, a hash operation is performed on the target address, and the characteristic information of the target address is converted into a numerical value as the mapping address. Then, using the generated mapping address, metadata, and target address, an initial record corresponding to the target IO request is generated; the key information of the target IO request is integrated in the initial record, and these information will play an important role in the subsequent processing flow, and are used to record and track the processing status and related data of the request in the distributed system.

[0069] S103. Obtain the status of the target record in each replica.

[0070] It should be noted that when inserting the initial record corresponding to the target IO request into the target nodes corresponding to each replica in the embodiment of the present application, a two-phase transaction, that is, a two-phase commit protocol, will be used for data writing.

[0071] Specifically, the two-phase transaction is a method for ensuring the atomicity, consistency, and isolation of transactions in a distributed system, and ensuring the reliable transmission and processing of data between multiple nodes. Considering that there are multiple replicas in the distributed system in the embodiment of the present application, when writing the metadata corresponding to each IO request for each replica, it is necessary to ensure the consistency of all replicas, that is, in order to ensure that the metadata corresponding to a certain IO request has been successfully written on each replica, the two-phase transaction protocol is used to coordinate the operations of the nodes where each replica is located.

[0072] Furthermore, the two-phase transaction has two phases. The first phase is the write request processing phase, and the second phase is the data persistence phase. In the write request processing phase, after the coordinator node in the distributed system receives the target data write request, it parses the request. That is, after obtaining the metadata and the target address, the coordinator then determines on which node of each replica the target record will be stored. That is, after determining the target nodes of each replica, it sends the write task to the corresponding target node. After each target node receives the task, it will maintain a status information for each record, and this status can reflect the progress of the record in the processing flow. Since the replicas are distributed on different nodes, by obtaining the record status in each replica, it can be determined whether the current record has been written to each replica to achieve data consistency and synchronization.

[0073] Specifically, the status of the target record may include the active state, the committed state, the aborted state, etc. After the target data is written, the status of the target record can be "committed state (commit)", which means the write has been completed; the status of the target record can also be "aborted state (abort)", then there may be an exception or other problems with the target node of a certain replica, and it cannot be written into the system; the status of the target record can be "active state (active)", which means the current record is being written.

[0074] S104. When the status of the target record is the committed state or the aborted state, adjust the data structure used to save the target record in the target node to a first-level pointer structure.

[0075] Among them, the first-level pointer structure includes the target record corresponding to the target IO request, as well as the metadata and the target address in the target record.

[0076] It should be noted that when the target record is in the committed state, it means that the write of the target record has been successfully completed, and the data has reached consistency among all replicas, that is, the current target record has been written to each replica. At this time, there is no need to read data using the target address and the mapped address, nor is it necessary to frequently dynamically adjust the data position, association information, etc. Therefore, the flexibility advantage brought by the second-level pointer structure is no longer obvious, and instead its complexity will increase the additional overhead.

[0077] When the target record is in the aborted state, it indicates that the operation has been terminated due to certain reasons (such as errors, conflicts, user cancellation, etc.). In this case, the second-level pointer structure corresponding to the current target record also loses its effect, and further simplifying the data structure can improve the efficiency of subsequent operations such as querying and cleaning the record.

[0078] Then, after determining that the target node is in the above two states, the original secondary pointer structure of the target record can be adjusted to a primary pointer structure, and the original mapped address can be replaced with the target address. Compared with the previous secondary pointer structure that needed to find the target address through the mapped address first, the primary pointer structure directly includes the metadata and the target address in the target record, reducing the step of addressing through the mapped address, thereby reducing the access latency and improving the data access speed. The secondary pointer structure requires additional space to store information such as the mapped address, while the primary pointer structure is more compact, reducing unnecessary memory occupancy and helping to improve the memory usage efficiency of the system. The simplification of the structure makes the management of the target record by the system more intuitive and easier. Whether it is for data cleaning, status update or other maintenance operations, the primary pointer structure can reduce the complexity of the operation and the probability of errors.

[0079] Refer to Figure 3 As shown, it is a schematic diagram of the adjusted target record in a node of the B+ tree structure. Among them, it includes the record linked list in node 1 of the tree structure. There are n records in this linked list. Through the KEY value corresponding to this record, the metadata target address corresponding to the current record is obtained.

[0080] Furthermore, when the present application receives a target data write request, this request corresponds to a target record. The system will copy this record to multiple replica nodes for processing. During the processing, each replica node uses a secondary pointer structure to save this record, which is convenient for flexible adjustment during data migration or update. When all replica nodes have completed the file write operation and the target record status becomes the committed state, the system will adjust the data structure for saving this record on each replica node from the secondary pointer structure to the primary pointer structure for more efficient subsequent access to this file.

[0081] Through the above method, in the target data write stage, the present application uses a secondary pointer structure to save the record corresponding to the target data. Furthermore, the mapped address can be flexibly updated through the secondary pointer structure, and the target address can be obtained through the mapped address, avoiding a large number of modifications to the target address corresponding to the record during data migration and improving the data maintenance efficiency. When the target record is in the committed or aborted state, it is converted to a primary pointer structure, and the metadata and target address corresponding to the target record are directly obtained through the primary pointer structure, reducing the addressing steps, reducing the access latency, accelerating the data access speed, and optimizing the overall data processing performance of the system.

[0082] Meanwhile, a tree structure is adopted to store records. Compared with array storage, nodes can reuse storage space, reduce the waste of space caused by scattered data storage, and improve the utilization rate of storage resources. In addition, by dynamically adjusting the pointer structure, after adjusting the secondary pointer structure to a primary pointer structure, the primary pointer structure is more compact than the secondary pointer structure, reducing unnecessary memory occupation and improving the memory usage efficiency of the system.

[0083] As an extension and refinement of the above embodiments, referring to Figure 4 as shown, the pointer dynamic conversion method for a distributed system provided by the embodiments of the present application further includes the following steps:

[0084] S401. After receiving a target IO request sent by a client, obtain the metadata and target address corresponding to the target IO request to generate a write request for the target data.

[0085] In some embodiments, when a client initiates a target IO request and the system receives the request, the request needs to be parsed and processed. By obtaining the metadata and target address corresponding to the request, a request for writing the target data is further generated to prepare for subsequent data storage operations.

[0086] S402. Generate a target record corresponding to the target IO request based on the target data, and insert the target record into a target node in each replica cache space.

[0087] Wherein, the cache space is set as a tree structure; the data structure for storing the target record in the target node adopts a secondary pointer structure; the secondary pointer structure includes the target record, the metadata and mapping address in the target record, and the target address pointed to by the mapping address.

[0088] Specifically, before generating the target record corresponding to the target IO request based on the target data and inserting the target record into the target node in each replica cache space, the embodiments of the present application further include the following steps 1 and 2:

[0089] Step 1. Create the tree structure in each replica respectively.

[0090] In the embodiments of the present application, a B+ tree structure is selected to be created according to system requirements and data characteristics. It is necessary to allocate corresponding memory space for the B+ tree structure on each replica node to implement the caching function.

[0091] It should be noted that in this application, the B+ tree structure is used to cache records corresponding to multiple IO requests. Each record has a unique persistent memory address for its metadata. Therefore, the key value in the record linked list of the B+ tree structure node can be set as the persistent memory address, and the corresponding record in the record linked list can be found through the key value (the persistent memory address of the metadata).

[0092] In the embodiment of this application, a tree structure is used to store records. Compared with using the current distributed system, when caching records, an array needs to be created in memory, and the maximum storage limit of the array is set to 1 million records. At the same time, the index of each record needs to be saved in the corresponding field, and the record status in the array is read through this field.

[0093] Compared with using an array to store records, the B+ tree is a multi-way balanced search tree that organizes data into nodes, and each node can store multiple key-value pairs. When storing metadata, the nodes of the B+ tree can efficiently reuse storage space. For example, a node can store the relevant information of multiple metadata, and these information are closely arranged, reducing the space waste caused by scattered data storage. Moreover, the height of the B+ tree is relatively low, and even when storing a large amount of metadata, it can maintain a good space utilization rate.

[0094] At the same time, the cache array usually needs to pre-allocate a fixed-size storage space, while the B+ tree can dynamically adjust the size of the node according to the actual amount of stored data. When the amount of data increases, the node can split to accommodate more data; when the amount of data decreases, the node can merge to release the redundant space. This dynamic adjustment mechanism enables the B+ tree to better adapt to different-scale data storage requirements and avoids excessive occupation of storage space.

[0095] Step 2: After receiving the write request for the target data, obtain the target node corresponding to the target record in the tree structure of each replica.

[0096] Specifically, when the system receives a write request for target data, it is necessary to first find the target node corresponding to the target record in the tree structure of each replica for subsequent accurate data writing. In the tree structure of each replica, the key value of the record to be written is compared with the key values of the records that have been written, and the target node is determined by comparing the key value ranges. If the leaf node is not full, the record is directly written to this leaf node; if the leaf node is full, a node split operation is performed, part of the key values and records are moved to a new node, and then the new record is written to the appropriate leaf node.

[0097] S403. Periodically traverse the records in each node of the tree structure in each replica to obtain the status of the target record in each replica.

[0098] Specifically, in a distributed system, when writing data to the system, the data will be stored on multiple replicas to enhance reliability and availability. The status of the target record on different replicas may vary due to factors such as network latency and node failures. By periodically traversing the record status of each node in the tree structure, the current status of the target record can be obtained, and subsequent operations can be determined based on the status of the target record in each replica.

[0099] At the same time, by periodically traversing the records in each node of the tree structure in each replica, status inconsistency problems can be detected in a timely manner, which helps to keep the data in all replicas consistent.

[0100] In some embodiments, a timing task can be set in the system to trigger the traversal operation of the nodes of the tree structure at a preset time interval (such as every minute, every hour, etc.).

[0101] S404. When the status of the target record is the committed state or the aborted state, adjust the data structure used to store the target record in the target node to a first-level pointer structure.

[0102] Wherein, the first-level pointer structure includes the target record corresponding to the target IO request, as well as the metadata and the target address in the target record.

[0103] S405. When the status of the target record is the active state, maintain the second-level pointer structure of the target node.

[0104] When the status of the target record is the active state, it indicates that the target record has not been written into the system yet, or there may be a problem with a certain replica, and data synchronization needs to be further performed through retry and rollback operations. Therefore, in this state, the existence of the second-level pointer structure enables the system to manage the target record more flexibly. The corresponding target address can be found through the mapping address in the second-level pointer structure, without worrying about the problem of the target address changing due to the deletion or migration of a certain record in the linked list. Furthermore, the system can efficiently complete various operations in the active state until the status of the target record changes to the committed or aborted state, and then perform optimization and adjustment of the data structure.

[0105] It should be noted that whenever the distributed system is powered off and restarted, the tree structure in each replica is rebuilt.

[0106] In a distributed system, each replica utilizes a tree structure (such as a B-tree, B+-tree, etc.) to efficiently manage and organize data. However, a system power outage can cause the data in memory to be lost, and the state of the tree structure will also be damaged. Therefore, when powering on and restarting, reconstructing the tree structure of each replica is to restore the system's data organization and management capabilities, ensuring that the system can operate normally and provide data services.

[0107] After the system restarts, after ensuring that the physical devices storing the data are not damaged, load the data that was previously persistently saved from the storage medium, that is, the information, metadata, key-value pairs, etc. corresponding to each node in the original tree structure. Then, allocate memory space for each replica and initialize the data structure of the tree structure. According to the type of tree structure previously used by the system, such as a B+-tree, create the corresponding root node and the basic framework of the tree structure. Insert the loaded data into the initialized tree structure node by node. During the insertion process, operations need to be performed according to the characteristics of the tree structure (such as the key-value sorting rules of nodes, the splitting and merging rules of nodes, etc.) to ensure the balance and orderliness of the tree structure.

[0108] Based on the same inventive concept, as an implementation of the above method, an embodiment of the present application also provides a pointer dynamic conversion device for a distributed system. This embodiment corresponds to the foregoing method embodiment. For the convenience of reading, the details of the foregoing method embodiment will not be elaborated one by one in this embodiment. However, it should be clear that a pointer dynamic conversion device for a distributed system in this embodiment can correspondingly implement all the contents of the foregoing method embodiment.

[0109] An embodiment of the present application provides a pointer dynamic conversion device for a distributed system. Figure 5 As shown in the structural schematic diagram of the pointer dynamic conversion device for the distributed system, Figure 5 as shown, the pointer dynamic conversion device 500 for the distributed system includes:

[0110] A receiving unit 501, configured to receive a write request for target data; the target data includes metadata corresponding to a target IO request and a physical address corresponding to the data requested by the target IO request;

[0111] An insertion unit 502, configured to generate a target record corresponding to the target IO request based on the target data, and insert the target record onto a target node in the cache space of each replica; the cache space is set as a tree structure; the data structure in the target node for storing the target record adopts a two-level pointer structure; the two-level pointer structure includes the target record, the metadata and mapping address in the target record, and the target address pointed to by the mapping address;

[0112] An acquisition unit 503, configured to acquire the status of the target record in each copy;

[0113] An adjustment unit 504, configured to, when the status of the target record is the committed status or the aborted status, adjust the data structure for storing the target record in the target node to a first-level pointer structure; the first-level pointer structure includes the target record corresponding to the target IO request, and the metadata and the target address in the target record.

[0114] As an optional implementation manner of an embodiment of the present application, the adjustment unit is further configured to acquire the status of the target record in each copy, and when the status of the target record is the active status, maintain the second-level pointer structure of the target node.

[0115] As an optional implementation manner of an embodiment of the present application, the acquisition unit 503 is specifically configured to periodically traverse the records in each node on the tree structure in each copy to acquire the status of the target record in each copy.

[0116] As an optional implementation manner of an embodiment of the present application, the receiving unit 501 is specifically configured to create the tree structure in each copy respectively; after receiving the write request of the target data, acquire the target node corresponding to the target record in the tree structure of each copy.

[0117] As an optional implementation manner of an embodiment of the present application, the pointer dynamic conversion method device for a distributed system further includes a reply unit, configured to reconstruct the tree structure in each copy whenever the distributed system is powered off and restarted.

[0118] As an optional implementation manner of an embodiment of the present application, the insertion unit 502 is specifically configured to generate the mapped address by using a hash algorithm based on the target address, and generate the target record based on the mapped address, the metadata, and the target address.

[0119] As an optional implementation manner of an embodiment of the present application, the receiving unit 501 is specifically configured to, after receiving the target IO request sent by the client, acquire the metadata and the target address corresponding to the target IO request to generate the write request of the target data.

[0120] Based on the same inventive concept, an embodiment of the present disclosure further provides an electronic device. Figure 6 The structural schematic diagram of the electronic device provided by the embodiment of the present disclosure is as Figure 6As shown in the figure, the electronic device provided in this embodiment includes: a memory 601 and a processor 602. The memory 601 is used to store a computer program. The processor 602 is used to execute the pointer dynamic conversion method for a distributed system provided in the above embodiment when executing the computer program.

[0121] Based on the same inventive concept, an embodiment of the present application also provides a computer-readable storage medium. A computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, the computing device is enabled to implement the pointer dynamic conversion method for a distributed system provided in the above embodiment.

[0122] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media that contain computer-usable program code.

[0123] The processor can be a central processing unit (CPU), or can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0124] The memory may include non-permanent memory in the computer-readable medium, random access memory (RAM), and / or non-volatile memory in the form of, for example, read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0125] Computer-readable media include both permanent and non-permanent, removable and non-removable storage media. The storage media can implement information storage by any method or technology, and the information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette tapes, disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information accessible by a computing device. As defined herein, computer-readable media do not include transitory media such as modulated data signals and carrier waves.

[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A pointer dynamic conversion method for a distributed system, characterized in that: Applied in distributed systems, including: Receive a write request for target data; the target data includes metadata corresponding to the target IO request and a physical address corresponding to the data requested by the target IO request; Generate a target record corresponding to the target IO request based on the target data, and insert the target record into the target node in each replica cache space; the cache space is set to a tree structure; the data structure for storing the target record in the target node adopts a secondary pointer structure; the secondary pointer structure includes the target record, the metadata and mapping address in the target record, and the target address pointed to by the mapping address; Get the status of the target record in each replica; When the status of the target record is a submitted status or an aborted status, the data structure in the target node for storing the target record is adjusted to a first-level pointer structure; the first-level pointer structure includes the target record corresponding to the target IO request, and the metadata and the target address in the target record.

2. The method according to claim 1, characterized in that: After generating the target record corresponding to the target IO request based on the target data and inserting the target record into the target node in each replica cache space, the method further includes: The state of the target record in each copy is obtained, and when the state of the target record is active, the secondary pointer structure of the target node is maintained.

3. The method according to claim 1, characterized in that: The obtaining the status of the target record in each copy includes: Periodically traverse the records in each node on the tree structure in each replica to obtain the status of the target record in each replica.

4. The method according to claim 1, characterized in that: Before generating an initial record based on the metadata and the target address and inserting the initial record into a plurality of target nodes, the method further includes: Creating the tree structure in each of the copies respectively; After receiving the write request for the target data, the target node corresponding to the target record is obtained in the tree structure of each replica.

5. The method according to claim 1, characterized in that The method further comprises: Whenever the distributed system is powered off and restarted, the tree structures in the respective replicas are rebuilt.

6. The method according to claim 1, characterized in that Before generating a target record corresponding to the target IO request based on the target data and inserting the target record into the target node in each replica cache space, the method further includes: Based on the target address, a hash algorithm is used to generate the mapping address, and based on the mapping address, the metadata and the target address, the target record is generated.

7. The method according to claim 1, characterized in that Before receiving the write request of the target data, the method further includes: After receiving the target IO request sent by the client, metadata and a target address corresponding to the target IO request are obtained to generate a write request for the target data.

8. A pointer dynamic conversion device for a distributed system, characterized in that: include: A receiving unit, used for receiving a write request of target data; The target data includes metadata corresponding to the target IO request and a physical address corresponding to the data requested by the target IO request; An inserting unit, configured to generate a target record corresponding to the target IO request based on the target data, and insert the target record into a target node in each replica cache space; the cache space is configured as a tree structure; The data structure used to store the target record in the target node adopts a secondary pointer structure; The secondary pointer structure includes the target record, the metadata and mapping address in the target record, and the target address pointed to by the mapping address; An acquisition unit, used for acquiring the status of the target record in each replica; An adjustment unit is used to adjust the data structure used to store the target record in the target node to a first-level pointer structure when the state of the target record is a committed state or an aborted state; the first-level pointer structure includes the target record corresponding to the target IO request, and the metadata and the target address in the target record.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory is used to store a computer program; and the processor is used to enable the electronic device to implement the pointer dynamic conversion method for a distributed system as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a computing device, the computing device implements the pointer dynamic conversion method for a distributed system as described in any one of claims 1 to 7.

Images