Index data persistence method and apparatus, storage medium, and electronic device

Abstract

The application relates to an index data persistence method and device, a storage medium and an electronic device. The method comprises the following steps: slicing index data in a memory to obtain a plurality of index shards; starting from when a last checkpoint operation is completed, persisting an index shard in a storage medium every time a total amount of M index data is written; and after a total amount of N index data is written and all the index shards are persisted in the storage medium, determining that the current checkpoint operation is completed, wherein N is the product of M and the number of index shards. The application solves the technical problem that index data persistence squeezes user requests.

Description

Technical Field

[0001] This application relates to the field of index persistence, and more particularly to an index data persistence method, apparatus, storage medium, and electronic device. Background Technology

[0002] In existing technologies, distributed systems involve the persistence of index data (metadata). However, persisting index data incurs significant CPU and bandwidth overhead, potentially disrupting normal user requests and negatively impacting user experience. Summary of the Invention

[0003] This application provides a method, apparatus, storage medium, and electronic device for persisting index data to solve the technical problem of index data persistence causing a surge in user requests.

[0004] In a first aspect, this application provides an index data persistence method, comprising: slicing index data in memory to obtain multiple index shards; starting from the completion of the checkpoint operation of the previous index data, persisting one of the aforementioned index shards to the storage medium for each M total amount of index data written; and determining that the current checkpoint operation is complete after writing a total amount of N index data and all the aforementioned index shards are persisted to the aforementioned storage medium, wherein the aforementioned N is the product of the aforementioned M and the number of the aforementioned index shards.

[0005] Secondly, this application provides an index data persistence apparatus, comprising: a processing module for slicing index data in memory to obtain multiple index fragments; a persistence module for persisting one of the index fragments to a storage medium for each M total amount of index data written, starting from the completion of the checkpoint operation of the previous index data; and a determination module for determining that the current checkpoint operation is complete after N total amount of index data has been written and all the index fragments have been persisted to the storage medium, wherein N is the product of M and the number of index fragments.

[0006] As an optional example, the above processing module includes: a sharding unit, used to shard the global index with a fixed size when the index data is a global index; or, when the index data is a global index and an incremental index, to shard the global index and the incremental index with fixed sizes respectively, wherein the fixed size is less than or equal to the size of the incremental index.

[0007] As an optional example, the above sharding unit includes: a sharding subunit, used to shard starting from the above global index, and add a sequence number to each of the above index shards; after the above global index sharding is completed, sharding the above incremental index, and continuing to add sequence numbers to the index shards of the above incremental index starting from the next to the last sequence number of the above global index, wherein the sequence number is used to indicate the persistence order of the above index shards.

[0008] As an optional example, the above persistence module includes: a persistence unit for locating the offset of the index fragment to be persisted via a foreground thread; copying the offset to a buffer, wherein after the offset is copied to the buffer, the index fragment in the memory is allowed to be accessed; and persisting the offset in the buffer to the storage medium.

[0009] As an optional example, the above apparatus further includes: a sending module, configured to send the start time information, persistence success information, and index shard location information of the current checkpoint operation to the cluster control node after determining that the current checkpoint operation is completed, so that the current node can read the current checkpoint by reading the start time information, persistence success information, and index shard location information from the cluster control node.

[0010] As an optional example, the above-mentioned device further includes: determining the value of M by the following formula: M>=0.929*Slice_Size / X(1) where Slice_Size is the size of each index slice and X is a preset value.

[0011] As an optional example, the above-mentioned device further includes: a rate limiting module, used to limit the user's read and write requests by adjusting the above-mentioned X, wherein the above-mentioned X represents the proportion of the bandwidth required to perform one checkpoint operation to the total bandwidth between two checkpoint operations, the larger the above-mentioned X is, the higher the rate limiting degree, and the smaller the above-mentioned X is, the lower the rate limiting degree.

[0012] Thirdly, this application provides an electronic device, comprising: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; and at least one memory connected to the at least one bus, wherein the memory stores a computer program, and the processor is configured to implement any of the above-described index data persistence methods when executing the computer program.

[0013] Fourthly, this application also provides a computer storage medium storing computer-executable instructions, which are used to execute the index data persistence method described in any of the above claims of this application.

[0014] Compared with the prior art, the technical solution provided in this application has the following advantages: The solution provided in this application obtains multiple index shards by slicing the index data in memory; starting from the completion of the checkpoint operation of the previous index data, each time a total of M index data is written, one of the above-mentioned index shards is persisted to the storage medium; after a total of N index data is written and all the above-mentioned index shards are persisted to the above-mentioned storage medium, it is determined that the current checkpoint operation is completed, where N is the product of M and the number of the above-mentioned index shards, thereby enabling the persistence of index data in time periods, reducing the congestion of user requests, and improving service quality. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0018] Figure 1 A flowchart illustrating an index data persistence method provided in this application embodiment;

[0019] Figure 2 This application provides an example of an index sharding method for persisting index data, illustrated in an embodiment of the present application.

[0020] Figure 3 A flowchart illustrating another index data persistence method provided in this application embodiment;

[0021] Figure 4 A flowchart illustrating yet another index data persistence method provided in this application embodiment;

[0022] Figure 5A flowchart illustrating yet another index data persistence method provided in this application embodiment;

[0023] Figure 6 A flowchart illustrating yet another index data persistence method provided in this application embodiment;

[0024] Figure 7 A schematic diagram of the structure of an index data persistence device provided in an embodiment of this application;

[0025] Figure 8 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] The following disclosure provides numerous different embodiments or examples for implementing various structures of the invention. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of the invention. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0028] To address the technical problem of index data persistence causing a surge in user requests in existing technologies, this application provides an index data persistence method that can reduce the surge in user requests and improve service quality.

[0029] Figure 1 This is a flowchart of an index data persistence method provided in an embodiment of this application.

[0030] like Figure 1 As shown, the above-mentioned index data persistence method includes:

[0031] S102, slice the index data in memory to obtain multiple index slices;

[0032] S104, starting from the completion of the checkpoint operation of the previous index data, each time a total of M index data is written, an index fragment is persisted to the storage medium;

[0033] S106. After writing a total of N index data and persisting all index shards to the storage medium, the checkpoint operation is determined to be complete, where N is the product of M and the number of index shards.

[0034] The aforementioned index data is used in a distributed system to record data pages. By using the index data, the address of a data page can be found, and then the data can be retrieved from that address. The index data is stored in memory, which is volatile; therefore, the index data can be persisted to a storage medium, such as a disk, for long-term storage.

[0035] In this embodiment, when persisting index data, the index data can be sliced ​​into multiple index shards, and then the index shards are persisted at intervals. The advantage of doing this is that splitting a large amount of data that would otherwise be persisted all at once, persisting a portion at a time, and having an interval between two persistences reduces the pressure on user requests and improves service quality.

[0036] When persisting index shards, starting from the time the last checkpoint operation was completed, each time the user writes a total of M data, one index shard is persisted. After multiple persistences, when the user has written a total of N data, N / M index shards (all index shards) have been persisted. Then the current checkpoint operation is complete. If index data in memory is lost, the persisted index shards can be moved into memory to recover the data pages.

[0037] Figure 2 This is a schematic diagram of index fragmentation in this embodiment. (The text appears to be incomplete and contains several typos. A more accurate translation would require the full context.) This represents the total bandwidth consumption between the start time of checkpoint epoch:n (the nth checkpoint) and the start time of checkpoint epoch:n+1 (the (n+1th checkpoint)). 0, 1, 2...# represent index slices, and Slice_Size is the size of the index slice, meaning the index data is sliced ​​to a fixed size. M GB refers to the total amount of data written, i.e., persisting an index slice to storage media such as disk.

[0038] The scheme provided in this application provides multiple index shards by slicing the index data in memory. Starting from the completion of the checkpoint operation of the previous index data, an index shard is persisted to the storage medium for every M total amount of index data written. After N total amount of index data has been written and all index shards have been persisted to the storage medium, the checkpoint operation is determined to be complete, where N is the product of M and the number of index shards. This allows for time-segmented persistence of index data, reducing the pressure on user requests and improving service quality.

[0039] As an optional example, such as Figure 3 As shown, slicing the index data in memory to obtain multiple index slices includes:

[0040] S302, when the index data is a global index, the global index is sharded with a fixed size; or, when the index data is a global index and an incremental index, the global index and the incremental index are sharded with fixed sizes respectively, wherein the fixed size is less than or equal to the size of the incremental index.

[0041] In this embodiment, there are several ways to slice the index data. When the index data is a global index, it can be sliced ​​into multiple index shards of a fixed size. When the index data includes both global and incremental indexes, it can also be sliced ​​into global and incremental index shards. Alternatively, the global index can be sliced ​​according to the size of the incremental index, resulting in both global and incremental index shards.

[0042] As an optional example, such as Figure 4 As shown, when the index data consists of a global index and an incremental index, sharding the global index and the incremental index with a fixed size includes:

[0043] S402, start sharding from the global index, and add a sequential number to each index shard after each shard is obtained;

[0044] S404, after the global index is sharded, the incremental index is sharded, and starting from the next sequential number of the last sequential number of the global index, sequential numbers are added to the index shards of the incremental index. The sequential numbers are used to indicate the persistence order of the index shards.

[0045] In this embodiment, the index shards for the global index and incremental index are ordered. If the index data only includes the global index, a number can be assigned to each fixed-size global index shard. However, if the index data includes both the global and incremental indexes, sequential numbers can be added to the global index shards first, followed by sequential numbers to the incremental index shards. Since the incremental index records the most recently changed data, its persistence order is later, reducing the impact of new data on the global index in memory.

[0046] As an optional example, such as Figure 5 As shown, starting from the completion of the checkpoint operation of the previous index data, each time a total of M index data is written, an index shard is persisted to the storage medium, including:

[0047] S502, locates the offset of the index shard that needs to be persisted through the foreground thread;

[0048] S504, copy the offset to the buffer, whereby after copying the offset to the buffer, the index fragment in memory is allowed to be accessed;

[0049] S506 persists the offset in the buffer to the storage medium.

[0050] In this embodiment, when persisting an index shard, the offset of the index shard in the global index is located from memory, and then the index data at that offset is copied to a buffer. After being copied to the buffer, the index shard in memory can continue to provide read and write services. That is, the index shard cannot be read or written only during the time the index shard is copied. Then, the offset in the buffer can be persisted to a storage medium, such as a disk, and the persistence process does not affect the read and write operations of the index shard in memory.

[0051] As an optional example, such as Figure 6 As shown, after confirming the completion of this checkpoint operation, the above method also includes:

[0052] S602 sends the start time information, persistence success information, and index shard location information of this checkpoint operation to the cluster control node, so that the current node can read the current checkpoint by reading the start time information, persistence success information, and index shard location information from the cluster control node.

[0053] In this embodiment, the start time information, persistence success information, and index shard location information of each checkpoint operation of each computing node are sent to the cluster control node, which stores or persists them to disk. If index data in the memory of any node is lost, the index data can be retrieved from the cluster control node, and the data page data can then be recovered.

[0054] As an optional example, before persisting an index fragment to storage media for each M-sized chunk of index data written, starting from the completion of the checkpoint operation of the previous index data, the method further includes determining the value of M using the following formula:

[0055] M >= 0.929 * Slice_Size / X

[0056] Where Slice_Size is the size of each index slice, and X is a preset value. In this embodiment, the read and write requests of users can also be rate-limited by adjusting X, where X represents the proportion of the bandwidth required for one checkpoint operation to the total bandwidth between two checkpoint operations. The larger X is, the higher the rate-limiting degree, and the smaller X is, the lower the rate-limiting degree.

[0057] In this embodiment, X can be preset. X indicates the percentage of bandwidth required for one checkpoint operation relative to the total bandwidth between two checkpoint operations. This setting can be used to rate limit user read and write operations. The larger the value, the higher the rate limiting level.

[0058] Here's an example. A checkpoint operation is performed on the index data every time a user writes a total of N GB of data. At the beginning of the checkpoint operation, the entire index data is sliced ​​into slices of a fixed size, `Slice_Size`. These slices are distributed evenly and sequentially between checkpoint intervals, with normal responses to user requests between slices. The interval between slice distributions is measured by the amount of data written to the node, and is set to M GB.

[0059] M = N / number of slices. That is, whenever a user writes M GB of data, a slice needs to be persisted. The persistence process allows the foreground thread to locate the offset of the slice to be persisted and copy it to the slice's dedicated flush buffer. Then, the foreground thread can respond to user requests normally, while the background thread persists the content in the buffer.

[0060] The determination of the values ​​of M and N involves the aforementioned preset value X. The value of X can be preset and can be any value, for example, set to 7.5%, which means that the bandwidth consumption required for one indexing data checkpoint operation accounts for no more than 7.5% of the total bandwidth consumption between two checkpoint operations.

[0061] The formula is:

[0062] Among them That is This represents the bandwidth consumption required for a single checkpoint operation on indexed data. This represents the total bandwidth consumption between two checkpoint operations.

[0063] Therefore, we can conclude that:

[0064] And M = N / #, and Where # represents the number of index slices, and Slice_Size represents the size of the index slice, therefore:

[0065] M >= 0.925 * Slice_Size / 7.5%

[0066] Given a slice size of Slice_Size, the data interval between slices should satisfy formula (4), that is, when a user writes M GB of data, a slice is persisted.

[0067] After calculating M, when a checkpoint operation begins, the entire index data is sliced ​​into slices of a fixed size (Slice_Size). The slice offset is initialized to 0, and the first slice is immediately persisted. The slice offset is then moved to the start of the next slice. When the user writes M GB of data after the previous slice's persistence, the slice indicated by the offset is persisted, and the offset is moved to the start of the next slice. This process continues until all slices from the checkpoint operation are stored on the storage node.

[0068] The start time of this checkpoint operation, the persistence success message, and its location on the storage node are sent to the cluster control node. The cluster control node persists this information using a distributed consistency protocol, thus completing a single checkpoint operation. This checkpoint operation is performed once every N GB of data written by the user.

[0069] When a compute node recovers from a power outage, it reads the latest checkpoint location and start time information from the cluster control node, and then reads the latest checkpoint from the storage node based on the location information. Based on the checkpoint start time information, it reads the index data written after the checkpoint start time from the storage node (i.e., the aforementioned index shards), recovers the corresponding index data, and merges it with the checkpoint. Once the index data recovery is complete, the compute node can process user requests normally.

[0070] Figure 7 This is a schematic diagram of the structure of an index data persistence device provided in an embodiment of this application. Figure 7 As shown, the above-mentioned index data persistence device includes:

[0071] Processing module 702 is used to slice the index data in memory to obtain multiple index slices;

[0072] The persistence module 704 is used to persist an index fragment to the storage medium every time a total of M index data is written, starting from the completion of the checkpoint operation of the previous index data;

[0073] The determination module 706 is used to determine that the current checkpoint operation is complete after writing a total of N index data and persisting all index fragments to the storage medium, where N is the product of M and the number of index fragments.

[0074] The aforementioned index data is used in a distributed system to record data pages. By using the index data, the address of a data page can be found, and then the data can be retrieved from that address. The index data is stored in memory, which is volatile; therefore, the index data can be persisted to a storage medium, such as a disk, for long-term storage.

[0075] In this embodiment, when persisting index data, the index data can be sliced ​​into multiple index shards, and then the index shards are persisted at intervals. The advantage of doing this is that splitting a large amount of data that would otherwise be persisted all at once, persisting a portion at a time, and having an interval between two persistences reduces the pressure on user requests and improves service quality.

[0076] When persisting index shards, starting from the time the last checkpoint operation was completed, each time the user writes a total of M data, one index shard is persisted. After multiple persistences, when the user has written a total of N data, N / M index shards (all index shards) have been persisted. Then the current checkpoint operation is complete. If index data in memory is lost, the persisted index shards can be moved into memory to recover the data pages.

[0077] Figure 2 This is a schematic diagram of index fragmentation in this embodiment. (The text appears to be incomplete and contains several typos. A more accurate translation would require the full context.) This represents the total bandwidth consumption between the start time of checkpoint epoch:n (the nth checkpoint) and the start time of checkpoint epoch:n+1 (the (n+1th checkpoint)). 0, 1, 2...# represent index slices, and Slice_Size is the size of the index slice, meaning the index data is sliced ​​to a fixed size. M GB refers to the total amount of data written, i.e., persisting an index slice to storage media such as disk.

[0078] The scheme provided in this application provides multiple index shards by slicing the index data in memory. Starting from the completion of the checkpoint operation of the previous index data, an index shard is persisted to the storage medium for every M total amount of index data written. After N total amount of index data has been written and all index shards have been persisted to the storage medium, the checkpoint operation is determined to be complete, where N is the product of M and the number of index shards. This allows for time-segmented persistence of index data, reducing the pressure on user requests and improving service quality.

[0079] For other examples of this embodiment, please refer to the examples above, which will not be repeated here.

[0080] like Figure 8 As shown in the figure, this application provides an electronic device, including a processor 111, a communication interface 112, a memory 113, and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.

[0081] Memory 113 is used to store computer programs;

[0082] In one embodiment of this application, when the processor 111 executes the program stored in the memory 113, it implements the index data persistence method provided in any of the foregoing method embodiments.

[0083] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the index data persistence method provided in any of the foregoing method embodiments.

[0084] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0085] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0086] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also include the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0087] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for persisting index data, characterized in that, include: Slice the index data in memory to obtain multiple index slices; Starting from the completion of the checkpoint operation of the previous index data, each time a total of M index data is written, one of the index fragments is persisted to the storage medium; After writing a total of N index data and persisting all index shards to the storage medium, the checkpoint operation is determined to be complete, where N is the product of M and the number of index shards. After determining that the current checkpoint operation is completed, the method further includes: sending the start time information, persistence success information, and index shard location information of the current checkpoint operation to the cluster control node, so that when the index data in the current node is lost, the current checkpoint can be read by reading the start time information, persistence success information, and index shard location information in the cluster control node, so as to recover the data page data. Specifically, starting from the completion of the checkpoint operation of the previous index data, before persisting an index fragment to the storage medium for each write of a total amount of M index data, the method further includes: determining the value of M using the following formula: M>=(1-X) Slice_Size / X Where Slice_Size is the size of each index slice, and X is a preset value; The method further includes: limiting the user's read and write requests through X, wherein X represents the proportion of the bandwidth required for one checkpoint operation to the total bandwidth between two checkpoint operations. The larger X is, the higher the rate limiting degree, and the smaller X is, the lower the rate limiting degree.

2. The method according to claim 1, characterized in that, The step of slicing the index data in memory to obtain multiple index slices includes: If the index data is a global index, the global index is sharded with a fixed size; or, When the index data consists of a global index and an incremental index, the global index and the incremental index are sharded with a fixed size, wherein the fixed size is less than or equal to the size of the incremental index.

3. The method according to claim 2, characterized in that, When the index data consists of a global index and an incremental index, the step of partitioning the global index and the incremental index into fixed-size shards includes: Starting from the global index, each index shard is sharded, and a sequential number is added to each index shard. After the global index is sharded, the incremental index is sharded, and starting from the next sequential number of the last sequential number of the global index, sequential numbers are added to the index shards of the incremental index, wherein the sequential numbers are used to indicate the persistence order of the index shards.

4. The method according to claim 1, characterized in that, Starting from the completion of the checkpoint operation of the previous index data, each time a total of M index data is written, the index fragment is persisted to the storage medium, including: The offset of the index shard that needs to be persisted is located through the foreground thread; The offset is copied into the buffer, wherein after the offset is copied into the buffer, the index fragment in memory is allowed to be accessed; The offset in the buffer is persisted to the storage medium.

5. An index data persistence device, characterized in that, include: The processing module is used to slice the index data in memory to obtain multiple index slices; The persistence module is used to persist an index fragment to the storage medium for each total M of index data written, starting from the completion of the checkpoint operation of the previous index data; The determination module is used to determine that the current checkpoint operation is completed after writing a total of N index data and persisting all the index fragments to the storage medium, wherein N is the product of M and the number of the index fragments; The device further includes a sending module, which, after determining that the current checkpoint operation is completed, sends the start time information, persistence success information, and index shard location information of the current checkpoint operation to the cluster control node, so that when the index data in the current node is lost, the current checkpoint can be read by reading the start time information, persistence success information, and index shard location information in the cluster control node, so as to recover the data of the data page. The device further includes a calculation module, configured to determine the value of M by the following formula, starting from the completion of the checkpoint operation of the previous index data and before persisting an index fragment to the storage medium for each write of a total amount of M index data: M>=(1-X) Slice_Size / X Where Slice_Size is the size of each index slice, and X is a preset value; The device further includes a rate limiting module, used to limit the user's read and write requests through X, wherein X represents the proportion of the bandwidth required for one checkpoint operation to the total bandwidth between two checkpoint operations. The larger X is, the higher the rate limiting degree, and the smaller X is, the lower the rate limiting degree.

6. An electronic device, characterized in that, include: At least one communication interface; At least one bus connected to the at least one communication interface; At least one processor connected to the at least one bus; At least one memory connected to the at least one bus, wherein the memory stores a computer program, and the processor executes the computer program to implement the index data persistence method according to any one of claims 1 to 4.

7. A computer-readable storage medium storing computer-executable instructions for performing the index data persistence method according to any one of claims 1 to 4 of this application.

Images